Advocate Remuneration Assistant
A question-answering system for Kenya's Advocates Remuneration Order and the court rulings that apply it, and a pre-registered test of what makes it accurate: the engineering around the model, or fine-tuning the model itself.
The live demo runs a hosted model (gpt-oss-20b on Groq), not the Qwen2.5-7B systems evaluated below. It is a prototype and not legal advice.
of 236 new test questions answered correctly by the full pipeline on Qwen2.5-7B-Instruct. Rulings from unseen cases score 78.3%.
when case scoping, long context snippets and the fee calculator are all removed. [−34.7, −21.2]
from QLoRA fine-tuning or DPO. On unseen rulings: −2.9 [−10.1, +4.9] and −2.9 [−12.5, +5.6] points.
What it does
The corpus is the Advocates Remuneration Order and a set of court rulings, held as 3,414 chunks. A question passes through five steps, and the model never does arithmetic.
- Hybrid retrieval. Dense vector search and BM25 keyword search, merged by reciprocal rank fusion.
- Case scoping. When a question names a case, retrieval is limited to that case.
- Context building. Retrieved chunks go to the model as long context snippets.
- Fee calculator. A router spots fee questions and computes the figure in code from the Order's schedule.
- Generation. The model answers and cites the chunks it used. The model backend is switched by environment variable.
Why the test can be trusted
An earlier 62-question study had been used while the pipeline was tuned, so it flattered the pipeline. The new test was built afterwards.
- The pipeline was frozen first (tag
pipeline-frozen-v1), then 276 candidate questions were drafted and every one was reviewed by hand: 194 kept, 42 rewritten, 40 dropped. - 14 rulings never used in training or earlier evaluation form an unseen stratum of 69 questions. The other strata are seen rulings (48), statute (23), no-answer (56) and fee calculation (40).
- Gold answers for the 40 fee calculations were computed by hand and compared with the calculator afterwards. All 40 agreed.
- Three different models played three roles: gpt-oss-20b wrote the training data, qwen/qwen3.8-27b wrote the questions and gpt-5.4-mini judges the answers.
- The hypotheses, the analysis plan and the set's hash (
95ddd72d65bffeca) were committed and tagged before any system under test was run.
Results
Pipeline parts are worth 5 to 28 points; fine-tuning cannot be told from zero
Change in overall accuracy against the system before the change, in percentage points, over 236 questions. Bars are 95% intervals.
Cluster bootstrap by case, 10,000 resamples. The identical rerun differed by 2 of 236 answers.
The pre-registered fine-tuning tests use unseen rulings only, where SFT and DPO both score 2.9 points lower than the system before them, inside intervals about 15 to 18 points wide.
| System | All | Unseen rulings | Seen rulings | Statute | No-answer | Calc |
|---|
Where the errors come from
On the 117 ruling questions, the gold evidence chunk was among the retrieved chunks 78.6% of the time for the base system (92 of 117), and it was missing for 14 of the 25 ruling questions the base system got wrong. Retrieval misses are a major source of error. They are not the only one: cutting snippets to 1,000 characters left recall unchanged at 78.6%, yet cost 26.1 points on unseen rulings. There the evidence was retrieved but, as far as these data show, the shortened snippet no longer carried the answer.
| System | Evidence recall, % | Faithfulness, mean |
|---|---|---|
| base | 78.6 | 4.59 |
| ep1 (SFT) | 78.6 | 4.59 |
| dpo | 78.6 | 4.47 |
| nocase | 67.5 | 4.47 |
| snip1000 | 78.6 | 4.41 |
| nocalc | 79.5 | 4.74 |
| stripped | 68.4 | 4.28 |
After the test: re-ranking retrieval
The table above pointed at retrieval, so after the pre-registered test I tried re-ranking: dense and keyword search each fetch 50 candidates instead of 5 and 3, and a cross-encoder (BAAI/bge-reranker-base) picks which ones fill the same seven slots in the prompt. This is exploratory. I chose it by looking at v3's failures, so the gain is in-sample, and both arms ran once, on different hardware from the pre-registered runs (2× T4, fp16), with the same code, settings and judge.
| Without | With | Change [95% CI] | |
|---|---|---|---|
| accuracy, all | 89.4% | 93.6% | +4.2 [+0.4, +8.2] |
| accuracy, unseen rulings | 75.4% | 88.4% | +13.0 [+4.1, +21.1] |
| false refusals, of 180 | 10 | 1 | |
| gold chunk found, of 140 | 80.0% | 90.0% |
Seventeen answers improved and seven got worse (sign test p = 0.064), so it is a promising single run rather than a settled result. About three of the seven losses came from the re-ranker crowding the context with near-duplicates, such as a neighbouring row of a fee table; the rest look like noise. A smaller, faster re-ranker and a cap on chunks per document both made retrieval worse. At about 14 seconds per question on a laptop CPU, the re-ranker stays off in the live demo; on a GPU it is fast.
Auditing this evaluation also turned up a retrieval metric that passed by construction on 72% of questions (171 of 236) and a judge that gave 7 of 10 wrong refusals a perfect score. The checks that caught them are packaged as a reusable Claude Code Agent Skill, rag-eval-harness, which adds the same kind of evaluation harness to any RAG app.
What each component costs in latency
The components that protect accuracy add almost no time. Switching the calculator off made answers slower, not faster, so the deterministic path is both more accurate and cheaper.
| Setting | Median | Mean | P95 |
|---|---|---|---|
| full pipeline | 2.10 | 2.39 | 4.83 |
| case scoping off | 2.03 | 2.04 | 3.98 |
| snippets cut to 1,000 chars | 1.83 | 2.04 | 4.54 |
| calculator off | 2.83 | 3.29 | 8.40 |
Shorter snippets save about 0.3 seconds on the mean and cost 11.9 points of accuracy, so that trade is not worth making. The calculator switch also turns off the guard, whose fixed replies skip the model entirely. With both off, fee questions and those guard-answered questions all go to the model, which is the likely reason for the slower answers and the tail at 8.4 seconds, though I did not time that split. Only about seven of the 40 questions are fee calculations, so the P95 rests on very few answers.
What the results show
Each pipeline component helps on a test it was not tuned on. Cutting snippets to 1,000 characters costs 26.1 points on unseen rulings. Turning the calculator off drops fee questions from 100% to 20%. Removing case scoping costs 4.7 points overall, the smallest effect.
Fine-tuning shows no detectable benefit at this sample size, and every interval includes zero.
What they do not show
That fine-tuning is useless. The unseen rulings come from only 14 cases, so a gain of a few points could hide in the intervals.
Statute questions cannot be truly unseen, and the judge is itself a model. I re-judged 30 randomly drawn answers by hand (seed 20261017) and agreed with the judge on all 30. Only 13 of those were scored by the model judge, the other 17 being deterministic checks on fee figures and refusals, so the real evidence is 13 for 13, which still allows a disagreement rate of about 20% or less at 95% confidence. One jurisdiction, one 7B model, one adapter of each kind.
The fee router and its guard also misfired: on 6 of the 236 questions (about 2.5%) they replied with a fixed fee message to a question that was not a fee calculation, so the full pipeline's accuracy is somewhat understated. I fixed both triggers after the test, and all six then answered correctly in a paired re-run, but that check is in-sample.
Reproduce it
The frozen pipeline, the 236 questions, the raw results for all eight configurations and the analysis script are in the repository.
export LLM_BASE_URL=https://<pod>-8000.proxy.runpod.net/v1 export LLM_API_KEY=<key> ./run_v3_eval.sh # refuses to run if the set's hash differs python eval/analyze_v3.py --out eval/v3_results.md