Your whole loop rides on one number

Your whole loop rides on one number

A score only drives the loop if you can trust it. Here is how the Evaluator’s judge keeps the score honest, and why most of that work happens before the model is even asked.

Ask a model to score an answer’s relevancy from 0 to 1, and run it three times on the same input: 0.73, then 0.61, then 0.80. Same answer, same model, nothing changed. The only thing that moved is the score, and that movement has a name: variance.

That variance would be harmless if the score were just a readout. It isn’t. The score is the input to every decision the loop makes next: what’s broken, what to change, whether the change worked. A wrong score doesn’t slow the loop down, it points it the wrong way, and a noisy score is wrong on a random schedule. So whether your agent improves at all comes down to whether you can trust one number through the noise. Here is what it takes to make that number honest, in order, and most of it happens before the model is even involved.

The grader can’t be the thing it grades

A model grading its own output is a student marking their own exam: the grade is fiction. The fix is structural. The thing that does the work and the thing that scores it are different processes, and the scorer never gets to treat its own reasoning as the answer key. Get this wrong and nothing downstream matters: the number is already cooked.

Stop scoring. Start checking.

“Answer relevancy: 0.73” tells you nothing you can act on; “cites a real source for each claim?” tells you exactly what’s broken. A rating drifts because there’s no fact of the matter to pin it down, so the model lands somewhere new each run, and that drift is the variance you saw at the top. A check is one concrete thing that’s either true or false. It can be wrong, which is exactly what makes it worth trusting, and when it fails it names the gap instead of docking a vibe by a point. So the first move is always to turn “how good is this?” into a list of yes/no questions a wrong answer could actually fail.

How the Judge actually decides

A check still has to be judged, and most of the noise people blame on “the model” is really the judge being allowed to skip steps. The Evaluator’s judge runs a fixed procedure, and every step is a place it can stop and say “can’t tell” instead of guessing.

Bind every word to the trace. Before it scores anything, the judge resolves every term in the criterion to something actually in the trace. If the criterion says “on-topic with the advertiser’s product” and the brief came in blank, there is no product to check against, so the honest result is “can’t tell,” not “fail.” A term with no referent is the single most common way a judge produces a confident wrong answer.

Evidence proves the claim, never the verdict. The judge can correctly prove “this copy is about tax software” and still be wrong to conclude “therefore it’s off-topic.” The proof is about the claim; the verdict smuggles in a premise nobody sourced (“tax software isn’t the advertiser’s product”). The gap between a true claim and the verdict it supposedly supports is where every confident-but-wrong judge lives. So a second, independent pass reads only that step and asks one question: does the claim actually entail the verdict? It can knock a verdict down to “can’t tell.” It can never talk one up.

No verdict without a receipt. The judge writes its critique and cites the exact field and value it’s leaning on before it is allowed to say pass or fail. Reverse that order and a model picks the answer first, then writes a story to justify it. Reasoning-first means the verdict has to follow from something on the page, and you can re-check the citation by exact match instead of taking its word.

Prove the absence. “The agent never called the tool” is the easiest thing in the world to hallucinate. Absence counts only when the judge points at the field where the call would have appeared and shows it empty. No positive check, no finding.

Pin the judge

Same model, temperature zero, recorded into the run. Then when the score moves, you know the agent changed, not the grader’s mood. An unpinned judge measures itself as much as it measures your agent, and after the fact you can’t tell the two apart.

Grade the grader

A judge isn’t trusted until it agrees with a human, on the good ones and the bad ones, measured on a set it was never shown while being built. Split your labeled cases into three piles that never touch: a handful to seed the judge’s own few-shot examples (two to four, with one borderline case, is plenty), one to tune against, and one sealed until the very end and opened once. Tune on the sealed set and you’re grading the judge on its own homework.

Two numbers matter here. Aim for 30 to 50 examples of each class, and keep the split balanced even when real failures are rare, or there’s nothing to calibrate the fail side against. And read agreement as two separate rates, how often it agrees on the passes and how often it agrees on the fails, never as one accuracy number. A judge that blindly says “pass” scores 95% accurate on a dataset that’s 5% failures, and catches none of them.

A check that flags everything measures nothing

A check that trips on your best run and your worst run alike measures nothing, and it has no business gating a release. So a check earns its place at the gate only when it fires on a known-broken run and stays quiet on a known-healthy one. Anything that can’t tell those two apart stays in the suite as a note, never as a gate.

Keep the score and its variance apart

A passing score from a high-variance judge isn’t a passing score. So the report carries two numbers that never get blended: the gate (did it pass) and the variance (how much the verdict moves when nothing changed). Average them into one health number and you’ve buried the single signal that tells you whether to believe the other: a 0.90 that swings 0.2 between runs is worth less than a steady 0.80.

The floor you can’t sand past

Even after all of this, a model judge still wobbles a little on identical input. There is a floor under how low its variance goes, and you don’t get past it. What the discipline buys you is reaching that floor instead of living far above it. So measure the variance, print it next to the score, and never celebrate a gain smaller than it. A win inside the noise band is just weather.

Field notes from building these

A few things that only show up once you’ve shipped one:

• Start around 50 to 100 traces. That’s roughly where new kinds of failure stop appearing. Fewer and you’re guessing at the distribution; more and you’re re-reading the same five bugs.
• Temperature zero is necessary, not sufficient. It kills sampling noise, not ambiguity. A vague rubric drifts at zero too.
• Measure the judge’s floor before you trust a number. Run it three to five times on identical input. If your “win” is smaller than that spread, you measured the weather, not an improvement.
• A check that has never failed is decoration. If a criterion hasn’t flagged a single real trace, it’s guarding nothing. Tighten it or retire it.
• “It called the tool” is not a pass. Sometimes the correct move is to do nothing. Grade the outcome, not the activity, or you reward a busy agent for being wrong.

Why it’s the whole game

Honest gate, compounding loop. Noisy gate, a very confident random number generator. Every cycle, your agent runs, the Evaluator scores it, the score says what to fix, you change something, and it goes again. A trustworthy score is the cheapest thing in that loop to build and the most expensive thing to skip: get it right and the loop gets a little better every pass; get it wrong and it doesn’t get smarter, it just gets louder.