Our LLM judge agreed with humans 61%: calibrating evals for an assistant
Published · Updated
We shipped an LLM-as-judge eval pipeline for a grounded assistant and trusted it for weeks. Then we hand‑labelled 200 cases and found the judge was wrong four times out of ten. Here is how we fixed it.
We run evals on every grounded assistant we ship, and for one client project we leaned hard on LLM‑as‑judge: a strong model scoring each answer for faithfulness and relevance against retrieved context. It felt rigorous. Dashboards were green, scores hovered around 0.9, and we used them to approve prompt changes. Then a domain expert reviewed a sample and disagreed with the judge constantly. We hand‑labelled 200 production traces and measured agreement: 61%. Barely better than a coin flip for borderline cases.
Why the judge was wrong
The failures clustered in three places. First, partially grounded answers: the response cited real retrieved text but added one unsupported claim, and the judge waved it through because most of it checked out. Second, domain vocabulary: the judge penalised correct answers that used client‑specific terminology it did not recognise. Third, verbosity bias: longer, well‑structured answers scored higher even when the extra content was ungrounded.
The calibration loop we built
We stopped treating the judge as ground truth and started treating it as a model that needs its own eval. We built a frozen human‑labelled set of 300 examples, deliberately oversampling the three failure clusters. Every judge prompt or model change now has to clear 85% agreement on that set before it can score production traffic. That single gate caught two later judge upgrades that would have quietly regressed scoring.
Splitting one score into three
Our original judge returned one faithfulness number. We split it into claim‑level checks: extract atomic claims from the answer, verify each against the retrieved context, then aggregate. Agreement with humans jumped from 61% to 84% on the calibration set. It costs about 2.3× more in judge tokens per evaluation, but claim‑level scores also tell us where answers fail, not just that they fail, which makes debugging prompts far faster.
Sampling beats full coverage
Running claim‑level judging on every production trace would have tripled our eval spend. Instead we score 5% of live traffic, stratified by intent type, plus 100% of traces flagged by cheap heuristics like low retrieval scores. That gives us drift detection within days at roughly 15% of the full‑coverage cost. The calibration set runs nightly in full, because that one is cheap and it guards the guard.
What we would tell anyone shipping evals
An LLM judge is a model, and models drift, get upgraded, and have blind spots. Build a small human‑labelled calibration set before you trust any automated score, oversample the cases where judge and humans are likeliest to disagree, and re‑validate the judge every time you change it. Two hundred labelled examples took our domain expert about six hours. That is the cheapest insurance in the entire pipeline.
Working on a project where these methods apply?