Evals are the testing machinery that keeps AI systems honest. This page explains them the way I wish someone had explained them to me: a pyramid you can draw from memory, metrics with real definitions, and the vocabulary in plain English. Everything here has a working implementation in Cherry, my open-source feedback-triage tool.
An eval is (a fixed set of test inputs) + (a scoring method) + (a pass/fail rule), run automatically on every change.
You can't skip them, because model systems fail quietly and in the wrong place. Edit line 40 of a prompt to fix one complaint, and something three features away gets worse with no error to tell you. Normal code fails loudly. Prompts just drift. Meanwhile the model underneath gets swapped every few months, like a factory where the machines replace themselves overnight. Without an eval, every prompt edit is a blind bet — and every model upgrade is a blind bet you didn't even place.
An eval answers one of two questions: did anything that used to work get worse (regression), or can it do the new thing at all (capability)? Separately, there are two kinds, and you need both. The contract eval asks whether the output is well-formed. The accuracy eval asks whether it's correct against human labels. A wrong-but-well-formed answer sails through the first and fails the second — like a beautifully filled-out form with the wrong diagnosis on it.
The example domain here is an AI-powered customer-feedback pipeline, but the pattern — an operational definition, a failure to watch, a gaming guard — transfers to any AI system you have to measure.
Median and p90 hours from record created to classified + owner assigned. Write down where the clock starts and stops, or every team measures it differently.
Watch — the p90 tail: "I submitted three weeks ago and heard nothing" is where trust dies, and the median hides it.
Two proxies: correction rate — % of AI classifications a human overrides, falling over time — and receiver usefulness — % of routed items the receiving owner one-taps as actionable.
Watch — one measures the machine, the other the whole pipeline. You need both to localize a problem.
% of shipped roadmap items whose spec cited a feedback-cluster ID at decision time. Companion pull metric: specs citing the system unprompted, per quarter.
Why hard — attribution (decisions have many parents), counterfactual (would've built it anyway), lag (quarters), gaming (post-hoc tagging — killed by the decision-time clause).
Repeat-submitter rate — % who submit again within 90 days — plus a 2-question pulse ("did you find out what happened?" / "would you submit again?").
Watch — behavioral beats survey: people don't keep feeding a system that eats their input.
users × runs/day × tokens × price is the bill; cost per triaged issue vs. the human-hour it replaces is the sentence that ends budget conversations.
Levers — prompt & result caching, model right-sizing, context trimming. Guards: per-user limits, alerts at 50/80/100%.
Per-field classification accuracy with confusion pairs (which boundary blurs tells you which definition to sharpen), and judge–human agreement ≥ ~90%.
Watch — an uncalibrated judge is a broken ruler: you optimize toward it anyway.
| Term | What it is | Say it like |
|---|---|---|
| regression eval | A fixed test panel re-run after every change (prompt edit, model upgrade). "Regression" means going backwards — the software sense, not statistics. Catching one here is a crash test at the factory; catching it in production is a recall. | "Did anything that used to be good get worse?" |
| CI | Continuous integration — the robot that runs your checks on every change, before merge. The badge is the live inspection certificate. | "The exam is administered automatically; no one has to remember." |
| golden set | Curated inputs with known-good outputs. Stack it with cases that burned you; every production miss becomes a candidate case. | "Standardized patients — actors with scripted, known conditions." |
| contract vs. accuracy | Contract: is the output well-formed (grounded, ranked, fields valid)? Accuracy: is it correct against human labels? | "Wrong-but-well-formed passes contract, fails accuracy." |
| LLM judge | A second model grading with a rubric, for judgment calls with no single right answer. Calibrate against human labels or it drifts lenient. | "Useful — and itself a model system with failure modes." |
| inter-rater reliability | The eval for the humans: two labelers grade the same sample independently; low agreement means the task definition is ambiguous — fix the guidelines, not the model. | "If trained humans agree 70% of the time, the taxonomy is the bug." |
| harness | Everything around the model: tool definitions, retries, schema enforcement, state, permissions, logging. Where agent quality mostly lives. | "Same model, better harness, dramatically better agent." |
| guardrail | Runtime enforcement that cannot be violated — schema enums, key isolation, rate caps. Guardrails catch the bad instance; evals catch the bad version. | "The form only has these checkboxes." |
| shadow mode | The new system runs silently alongside the old; outputs are compared, never shipped. Zero blast radius. | "Where trust data comes from before anyone is exposed." |
| staged rollout | Shadow → 5% → 25% → 100%, gated on metrics. Prompts as versioned config, so rollback is a pointer flip, not archaeology. | "A/B infrastructure, applied to prompts." |
| drift | Inputs shift under you, or the model gets swapped (upgrade). There is no permanent "qualified" state. | "Every model upgrade is a new machine on the line — requalify." |
| prompt injection | Untrusted text (a scraped review, a pasted ticket) trying to become instructions. A harness problem — delimit, never grant authority, gate consequential actions. | "You can't prompt your way out; the harness has to hold." |
| default-deny | Records with unknown permissions are treated as most restrictive. The failure mode becomes "overly careful," never "used what we shouldn't." | "Terms travel with the data as metadata, enforced by the pipeline." |