LLM as a Judge: The Eval Method Everyone Uses and Nobody Calibrates

LLM as a judge means using one language model to score the output of another against criteria you define. It has become the default way teams run LLM evals on free-text tasks, because reference metrics like BLEU and ROUGE measure word overlap rather than meaning, and human review does not scale. The method works. The problem is that most teams deploy a judge without calibrating it or measuring its noise, then make ship decisions on score movements the judge cannot actually resolve. This page covers how the method works, the documented biases, and the noise that quietly undermines it.

What Is LLM as a Judge?

With LLM as a judge, a model is given an evaluation prompt instead of a generation task. It reads an output, applies your criteria, and returns a score or label. Three formats dominate. Pairwise comparison asks which of two outputs is better. Direct scoring grades a single output against criteria such as correctness or tone. Reference-based evaluation compares the output to a gold answer or a retrieved context.

The approach was popularized by the MT-Bench work of Zheng et al. (2023), where a strong judge model reached over 80 percent agreement with human preferences, a level the authors compared to agreement between two humans. That result is why the technique spread from research into production pipelines. It scales where human annotation cannot, and it grades meaning where string-overlap metrics grade phrasing.

A minimal judge is easy to picture. The evaluation prompt states the task, the criteria, and the scale, then presents the question, the gold answer if one exists, and the candidate output. The judge returns a verdict and a short justification. Everything that makes LLM as a judge work in production is refinement of that skeleton, because the skeleton alone drifts with wording, order, and length in ways the next two sections quantify.

Which LLM Evaluation Metrics Should You Use?

Split your metrics by whether the task has a checkable answer. Deterministic checks are cheaper, faster, and noise-free, so use them everywhere they fit. Model-graded metrics earn their cost only on free-text quality.

Task shape	Metric type	Examples
Single correct answer	Deterministic	exact match, numeric tolerance, schema validation
Discrete label	Deterministic	classification accuracy, pass rate
Behavior constraints	Deterministic	regex checks, banned-content filters, length bounds
Free-text quality	LLM judge	correctness vs a gold answer, faithfulness, helpfulness
Retrieval-augmented output	LLM judge	faithfulness to context, answer relevancy
Agent trajectories	LLM judge + trace checks	task completion, tool-call correctness

A practical rule for LLM testing pipelines: every metric that can be deterministic should be, and the judge handles the remainder. Mixed suites also give you a sanity check, because a judge score that moves while every deterministic metric holds still deserves suspicion.

Cost shapes the split too. A deterministic check costs microseconds and nothing. Every LLM as a judge call costs a model invocation, and a 500-item suite scored on three criteria is 1,500 judge calls per run. Teams that put the judge everywhere stop running their evals within a month. Teams that reserve it for the questions only a judge can answer run their llm evals on every commit, and the eval you run beats the thorough one you skip.

What Biases Do LLM Judges Have?

The research record on judge bias is extensive, and the Wikipedia summary of the field catalogs the recurring ones. Position bias makes judges favor whichever answer is presented first. Verbosity bias makes them prefer longer outputs, in some measurements more than 90 percent of the time when content is equivalent. Self-preference makes a judge rate outputs from its own model family higher. Judges are also sensitive to formatting, paraphrasing, and option order, and the literature generally concludes they should augment human evaluation rather than replace it.

Standard mitigations exist for each. Swap answer order and average the two runs. Instruct the judge to ignore length. Use a judge from a different model family than the system under test. These fixes are cheap, and skipping them is how a plausible-looking eval quietly measures presentation instead of quality.

How Noisy Is an LLM Judge?

Bias is only half the reliability problem. The other half is variance, and it survives even at temperature zero. Score the same output with the same judge on the same dataset three times and you get three different numbers, because the judge samples its verdict rather than computing it. That spread is why a small reported gain can be nothing but re-evaluation noise, and it is why an eval reports a threshold before it reports a delta.

Two consequences are worth internalizing. First, calibration shrinks the spread, anchored rubrics and a pinned temperature help, but it never reaches zero. Second, real improvements on free-text tasks are often small, so they land inside the very band where judge noise lives, which is exactly why an uncalibrated pipeline cannot tell a genuine gain from a re-score of the same prompt. Measure your own spread before you trust any delta, and treat that spread, not zero, as the bar a real change has to clear.

How Do You Build an LLM Judge That Holds Up?

The reliable recipe is short, and every step exists to remove a known failure mode.

1. Build the dataset from real traffic, not synthetic prompts. Failures from production traces make the eval measure what actually breaks.
2. Hand-label a sample first. Your labels define the criteria and later tell you whether the judge agrees with you.
3. Use binary or low-precision scales. Judges grade “acceptable or not” far more consistently than a 1 to 10 score.
4. Anchor every tier with worked examples. Ship each score tier with a sample question, gold answer, and graded candidate. Anchors turn hard absolute judgments into graded similarity comparisons, which a judge handles far more reliably than a bare numeric scale.
5. Pin temperature to zero and request a reason alongside each score, so failures can be audited.
6. Measure your own judge noise. Re-run the unchanged baseline several times and record the spread. That spread, not zero, is the threshold a real change must clear.

The last step is the one that almost nobody does, and it is also the cheapest. Skipping it does not just risk one bad call. Missing evaluation and testing is one of the most common failure modes builders report, and Mutagent’s community-research methodology walks through how that pattern shows up at field scale.

How Do You Run LLM Evals in CI?

An eval that runs on demand measures curiosity. An eval that runs on every change measures the system, so the end state for llm evaluation is the same as for tests: wired into CI with a pass condition a machine can enforce.

The wiring has three parts. First, a stored baseline, the scored results of the current production configuration on the golden set. Second, a trigger list, since prompt edits, model swaps, retrieval changes, and tool changes all warrant a run even when no code changed. Third, a gate rule that uses your measured judge noise as the threshold: a candidate that scores below baseline by more than that band blocks the change, a candidate inside the band ships as neutral, and a candidate above it by more than the band is a real improvement worth recording as the new baseline.

Two operational details keep the pipeline honest over time. Pin the judge model version, because a silent judge upgrade shifts scores across the board and reads as a regression that never happened. And log every judge verdict with its reason, so that when a gate fires, the failure review starts from evidence instead of from a bare number.

What Makes LLM as a Judge Fail Silently?

The failure stories repeat, and knowing them is cheaper than rediscovering them. A team A/B tests two prompts with an uncalibrated judge and ships the loser, because position bias outweighed the quality difference. A team celebrates a two-point weekly gain that a re-run of the unchanged baseline would have shown to be noise. A team lets the golden set age for a quarter, and the LLM as a judge pipeline keeps passing while production quality drops, because the questions users now ask are no longer in the set.

The common thread is that nothing errors. Every one of these pipelines ran green while measuring the wrong thing, which is why the calibration steps above are not polish but the difference between an eval and a random number generator with a dashboard.

How Is Agent Evaluation Different?

Agent evaluation grades a trajectory, not a completion. An agent picks tools, loops, and delegates, so the artifact under review is the full sequence of decisions, and an LLM as a judge setup for agents needs that sequence as input. In practice that means agent evaluation depends on tracing, because without a span-level record of tool calls there is nothing trustworthy to grade.

The grading itself splits cleanly. Deterministic checks handle what they can, such as whether the right tool was called with valid arguments and whether the task ended within budget. The judge handles the rest, such as whether the plan was sensible and the final answer correct. Everything above still applies, including judge noise, and RAG pipelines add their own metric layer on top.

Mutagent runs this loop as a product. Our agents build eval sets from your traces, score them with calibrated judges, and gate changes on deltas that clear the measured noise, as part of the agent lineup you can point at your own system.