Portfolio project · Mike Keror · October 2026

Advocate Remuneration Assistant

A question-answering system for Kenya's Advocates Remuneration Order and the court rulings that apply it, and a pre-registered test of what makes it accurate: the engineering around the model, or fine-tuning the model itself.

The live demo runs a hosted model (gpt-oss-20b on Groq), not the Qwen2.5-7B systems evaluated below. It is a prototype and not legal advice.

87.7%

of 236 new test questions answered correctly by the full pipeline on Qwen2.5-7B-Instruct. Rulings from unseen cases score 78.3%.

−28.0 pts

when case scoping, long context snippets and the fee calculator are all removed. [−34.7, −21.2]

No gain

from QLoRA fine-tuning or DPO. On unseen rulings: −2.9 [−10.1, +4.9] and −2.9 [−12.5, +5.6] points.

What it does

The corpus is the Advocates Remuneration Order and a set of court rulings, held as 3,414 chunks. A question passes through five steps, and the model never does arithmetic.

  1. Hybrid retrieval. Dense vector search and BM25 keyword search, merged by reciprocal rank fusion.
  2. Case scoping. When a question names a case, retrieval is limited to that case.
  3. Context building. Retrieved chunks go to the model as long context snippets.
  4. Fee calculator. A router spots fee questions and computes the figure in code from the Order's schedule.
  5. Generation. The model answers and cites the chunks it used. The model backend is switched by environment variable.

Why the test can be trusted

An earlier 62-question study had been used while the pipeline was tuned, so it flattered the pipeline. The new test was built afterwards.

Results

Pipeline parts are worth 5 to 28 points; fine-tuning cannot be told from zero

Change in overall accuracy against the system before the change, in percentage points, over 236 questions. Bars are 95% intervals.

Cluster bootstrap by case, 10,000 resamples. The identical rerun differed by 2 of 236 answers.

The pre-registered fine-tuning tests use unseen rulings only, where SFT and DPO both score 2.9 points lower than the system before them, inside intervals about 15 to 18 points wide.

Accuracy, % of questions correct. The lowest score in each column is in bold.
SystemAllUnseen rulingsSeen rulingsStatuteNo-answerCalc

Where the errors come from

On the 117 ruling questions, the gold evidence chunk was among the retrieved chunks 78.6% of the time for the base system (92 of 117), and it was missing for 14 of the 25 ruling questions the base system got wrong. Retrieval misses are a major source of error. They are not the only one: cutting snippets to 1,000 characters left recall unchanged at 78.6%, yet cost 26.1 points on unseen rulings. There the evidence was retrieved but, as far as these data show, the shortened snippet no longer carried the answer.

Ruling questions only (n = 117). Evidence recall: the gold evidence chunk was among the retrieved chunks. Faithfulness: the judge's 1 to 5 score, mean. Descriptive only; no intervals computed.
SystemEvidence recall, %Faithfulness, mean
base78.64.59
ep1 (SFT)78.64.59
dpo78.64.47
nocase67.54.47
snip100078.64.41
nocalc79.54.74
stripped68.44.28

After the test: re-ranking retrieval

The table above pointed at retrieval, so after the pre-registered test I tried re-ranking: dense and keyword search each fetch 50 candidates instead of 5 and 3, and a cross-encoder (BAAI/bge-reranker-base) picks which ones fill the same seven slots in the prompt. This is exploratory. I chose it by looking at v3's failures, so the gain is in-sample, and both arms ran once, on different hardware from the pre-registered runs (2× T4, fp16), with the same code, settings and judge.

Base model, 236 v3 questions, paired runs that differ only in re-ranking. False refusals: answerable questions answered “Not covered”. Gold chunk found: the passage holding the answer was retrieved. Interval: cluster bootstrap by case, 10,000 resamples.
WithoutWithChange [95% CI]
accuracy, all89.4%93.6%+4.2 [+0.4, +8.2]
accuracy, unseen rulings75.4%88.4%+13.0 [+4.1, +21.1]
false refusals, of 180101
gold chunk found, of 14080.0%90.0%

Seventeen answers improved and seven got worse (sign test p = 0.064), so it is a promising single run rather than a settled result. About three of the seven losses came from the re-ranker crowding the context with near-duplicates, such as a neighbouring row of a fee table; the rest look like noise. A smaller, faster re-ranker and a cap on chunks per document both made retrieval worse. At about 14 seconds per question on a laptop CPU, the re-ranker stays off in the live demo; on a GPU it is fast.

Auditing this evaluation also turned up a retrieval metric that passed by construction on 72% of questions (171 of 236) and a judge that gave 7 of 10 wrong refusals a perfect score. The checks that caught them are packaged as a reusable Claude Code Agent Skill, rag-eval-harness, which adds the same kind of evaluation harness to any RAG app.

What each component costs in latency

The components that protect accuracy add almost no time. Switching the calculator off made answers slower, not faster, so the deterministic path is both more accurate and cheaper.

Seconds per answer for the pipeline call only, no judging. 40 questions (every sixth, all strata), base model on one vLLM server, single run after 3 untimed warm-up questions. No repeat run, so differences of a few tenths of a second may be noise.
SettingMedianMeanP95
full pipeline2.102.394.83
case scoping off2.032.043.98
snippets cut to 1,000 chars1.832.044.54
calculator off2.833.298.40

Shorter snippets save about 0.3 seconds on the mean and cost 11.9 points of accuracy, so that trade is not worth making. The calculator switch also turns off the guard, whose fixed replies skip the model entirely. With both off, fee questions and those guard-answered questions all go to the model, which is the likely reason for the slower answers and the tail at 8.4 seconds, though I did not time that split. Only about seven of the 40 questions are fee calculations, so the P95 rests on very few answers.

What the results show

Each pipeline component helps on a test it was not tuned on. Cutting snippets to 1,000 characters costs 26.1 points on unseen rulings. Turning the calculator off drops fee questions from 100% to 20%. Removing case scoping costs 4.7 points overall, the smallest effect.

Fine-tuning shows no detectable benefit at this sample size, and every interval includes zero.

What they do not show

That fine-tuning is useless. The unseen rulings come from only 14 cases, so a gain of a few points could hide in the intervals.

Statute questions cannot be truly unseen, and the judge is itself a model. I re-judged 30 randomly drawn answers by hand (seed 20261017) and agreed with the judge on all 30. Only 13 of those were scored by the model judge, the other 17 being deterministic checks on fee figures and refusals, so the real evidence is 13 for 13, which still allows a disagreement rate of about 20% or less at 95% confidence. One jurisdiction, one 7B model, one adapter of each kind.

The fee router and its guard also misfired: on 6 of the 236 questions (about 2.5%) they replied with a fixed fee message to a question that was not a fee calculation, so the full pipeline's accuracy is somewhat understated. I fixed both triggers after the test, and all six then answered correctly in a paired re-run, but that check is in-sample.

Reproduce it

The frozen pipeline, the 236 questions, the raw results for all eight configurations and the analysis script are in the repository.

export LLM_BASE_URL=https://<pod>-8000.proxy.runpod.net/v1
export LLM_API_KEY=<key>
./run_v3_eval.sh                       # refuses to run if the set's hash differs
python eval/analyze_v3.py --out eval/v3_results.md