Create evals from production failures and build a trustworthy eval suite
Mutagent derives custom LLM-as-a-judge & code-based graders from production failures, and calibrates them for internal consistency and with human expert feedback.
Available today as a guided design-partner engagement.
You can measure the agent. But do you trust your evals?
A generic LLM-as-a-judge does not measure what good means and scores differently each run, so you measure noise.
From production failures to a trustworthy eval suite
Mutagent derives the graders, calibrates them, then runs and grows the suite.
Analyzes your traces
*discover-evalsIt reads your history and system prompt to derive evals.
Learn the rules
It pulls must and never rules from your traces into checkable criteria.
Calibrate to your domain experts
A trace review aligns the LLM review with your experts to match their judgement.
Evaluate impact of every change
Run trusted regression & quality evals on every change.
Continuously expand test coverage
Expand your eval suite with failures surfaced from production.
Create your eval suite from production failures
Your production traces and failures become a labelled dataset with checkable criteria and stable scores.
Calibrate the graders to your domain experts
A short expert trace review aligns the judge with your team, so it stays internally consistent and trusted where it matches them.
Reads the traces you already collect.
Point it at your observability stack. It turns those traces into a bar you can trust.
Langfuse
LangSmith
OpenTelemetry
Braintrust- A calibrated judge
- A pass or fail scorecard
- Free deterministic checks
- A merge gate
What you can do with it
Catches real failures
The bar is learned from your production traces.
Same score twice
A pinned judge gives the same verdict every run.
No rubric to write
The rules come out of your traces, not a blank page.
Free at scale
Every finding becomes a check that runs for free.
The Evaluator vs A raw LLM-as-judge
Both point a model at your output and return a score. The difference is whether that score means the same thing twice, and whether it matches what your team actually calls quality.
Questions
How is this different from an LLM-as-judge I could set up myself?
A bare judge scores the same output differently every run and drifts on phrasing, so you cannot separate a real quality gain from random noise. The Evaluator pins the judge to one model at a fixed temperature and calibrates it to your team's ground-truth labels, so it reproduces run to run and matches what you call quality. Criteria are mined from your traces rather than hand-authored, and every verdict is backed by cited trace evidence.
Do I have to write the rubric first?
No. That is the wall most teams never get past. The Evaluator reads your traces and system-prompt rules to extract the MUST and NEVER behaviors and codifies them into checkable criteria. You get a working starting frame on day one instead of a blank page. It is a seed that compounds as your traces grow, not a finished rubric you have to author upfront.
How does it stay reproducible if models are non-deterministic?
It makes the diagnosis reproducible, not the underlying model. The judge is pinned to a recorded model id at a fixed temperature, and run identifiers, timestamps, and paths are masked before comparison, so re-running the same traces returns the same gate every time. That gives you a stable bar to verify a fix against. It does not make a non-deterministic model deterministic.
What does the verdict actually look like?
A severity-gated verdict per behavior: fail, incomplete, or pass. Each score is paired with the trace behind it, so you see the behavior that was scored, the criterion it missed, and the message that proves it. A behavior passes only when it has zero critical or high failures, so a minor drift never green-washes a real regression up to pass.
Is running a judge on all my traces going to cost a fortune?
No, because the judge does not run on every trace forever. The first time it finds a failure pattern, that pattern becomes a deterministic code check that runs on all future traces at no model cost. Expensive judge runs are for discovery; the free checks handle enforcement. Coverage grows while the marginal cost falls.
Is the judge accurate, or does it just look consistent?
Calibration to your labels comes first, because consistency without accuracy would be useless. The judge is calibrated against a reliability benchmark to your team's ground-truth labels, and both reproducibility and agreement with those labels are measured before it is trusted. An independent verify step re-resolves every cited reference to kill false-greens, so a verdict is never fabricated when a reference does not resolve.
What does the Evaluator not do?
It is the auditor, never the optimizer. It does not change your prompts, that is Build and Improve. It does not fine-tune model weights, that is the downstream Improve loop. It analyzes traces but does not produce them, that is your observability layer upstream. It measures whether hard guardrails exist and flags their absence as a critical defect, but it does not build them. It grades a fixed agent-and-prompt snapshot and hands its failures to Diagnose.
Where does it sit in the lifecycle?
At the Evaluate stage, after Build and downstream of Spec. Its failures feed Diagnose, and its gate is the go or no-go bar the Improve loop has to clear before shipping. Because it grades a fixed agent-and-prompt snapshot the same way twice, it is the stable bar the rest of the lifecycle measures against.
Score every change against what good really means.
Book a custom demo and we will calibrate the judge on your traces.