02 · Evaluate & testComing soon
Evaluate
Builds the evals and a calibrated judge that scores every change
Derives eval criteria from your historic traces and domain expertise, then builds and calibrates the judge against expert ground truth so its scores match what your team calls quality. Every other agent scores against this framework.
The Evaluator is the central authority in the Mutagent loop. Experiment Agent ranks against it. Mutation Agent validates against it. Monitoring / Incident Agent scores live traffic against it. A signed evidence pack from the Evaluator is what unlocks the merge button.
What it does
- Derives criteria from traces and domain expertise
- Builds agent-level evals or per-key rubrics
- Calibrates the judge against expert ground truth
Inputs
- Historic traces
- Domain expertise
- Expert ground truth labels
Outputs
- Calibrated evaluator with agreement scores
- Per-key rubrics and deterministic code checks
- Per-criterion drift detection
Works with
Braintrust
Phoenix
MLflow
GitHubGet early access to Evaluate
Join the early-access list and we will reach out the moment this agent ships.