Did the change make it better?
The average says yes. Eleven cases that used to pass now fail — those are the reason to hold, and an average score cannot show them to you.
Each case, run 10 times
One block = one run. Same input every time.A case that passes 10 out of 10 and a case that passes 9 out of 10 are different products. Most tools show the last run and render them identically — which is how intermittent failures reach customers.
Where did it go wrong?
Every step here looks correct on its own. The agent retrieved the wrong customer's record at step 4, then reasoned flawlessly about the wrong person for the next 58 steps.
Where the graders don't agree
A human and a model reviewed the same answer and reached opposite conclusions. That means one of three things — the model judge is miscalibrated, the human is wrong, or the rule itself is ambiguous. All three are worth fixing, so Verdict never averages them into one score.
Eval sets
A set everything passes is not a good set — it is a set that stopped telling you anything. Verdict flags saturation instead of celebrating it.
| Set | Cases | Runs each | Pass rate | Last graded |
|---|---|---|---|---|
| support-triage-160active | 160 | 10 | 84.1% | 12 min ago |
| billing-edge-40 | 40 | 10 | 61.0% | 2 days ago |
| tone-hard-25 | 25 | 20 | 48.4% | 2 days ago |
| smoke-12saturated | 12 | 5 | 98.3% | 12 min ago |
smoke-12 has passed above 95% for six consecutive weeks. It no longer separates a good agent from a bad one — it only confirms the service is up. Verdict suggests retiring it or adding harder cases rather than letting it inflate the aggregate.
What "correct" currently means
Changing a rule changes what every score means. Verdict keeps old scores readable but refuses to compare across versions — a comparison between two definitions of correct is not a comparison.
Clauses in effect
Agreement is measured per clause.A response passes if it resolves the customer's request without breaking policy.
Tone should suit the context of the incoming message.
A response that resolves one of several requests passes only if it states clearly what is not being handled and why.
Partial identifiers may be repeated back only where the customer's own jurisdiction permits it.
Version history
What grading costs
Unpredictable spend is why teams stop running evals, and a tool nobody runs produces no quality signal. Every action shows its estimated cost before you click it, not on a bill at the end of the month.
Eval runs per engineer, per week
Cost estimates shipped in week 3.Showing the cost made people run more evals, not fewer. The uncertainty was the deterrent, not the price.
What each action costs
| Action | Scope | Estimate |
|---|---|---|
| Grade one case | 10 runs, 1 model grader | $0.026 |
| Grade remaining ungraded | 18 cases | $0.47 |
| Re-grade full set | 160 cases, current rubric | $4.16 |
| Re-grade after rubric change | 160 cases, all 4 clauses | $8.90 |
How is the agent doing?
Production quality and evaluation quality are different questions. This page never blends them into one number — an agent can pass every test and still fail customers.
Recent activity
Needs a decision
Everything the agent did in production
Logs are runs that reached a real customer. They are graded on a sample, not in full — grading every production run would cost more than the agent itself.
| Time | Input | Steps | Latency | Grade | Cost |
|---|---|---|---|---|---|
| 14:02:11 | Refund not received after 14 days | 18 | 4.1 s | Passed | $0.014 |
| 14:01:58 | Change address on shared account | 62 | 12.7 s | Failed | $0.061 |
| 14:01:40 | Where is my order — tracking stuck | 14 | 3.3 s | Passed | $0.011 |
| 14:01:22 | Cancel subscription, wants retention offer | 27 | 6.8 s | Not graded | $0.023 |
| 14:00:55 | Two questions in one message | 31 | 7.4 s | Failed | $0.029 |
| 14:00:31 | Damaged item, wants replacement | 16 | 3.9 s | Not graded | $0.013 |
| 14:00:09 | Duplicate charge on card | 22 | 5.2 s | Passed | $0.019 |
The 62-step run at 14:01:58 is the one worth opening. Step count is the cheapest signal there is — an agent that needed three times the usual steps almost always went somewhere it should not have.
Is it getting worse?
A quality drop and an input-mix change look identical on a line chart. Verdict separates them before raising an alert, because the fix is completely different.
Production pass rate, sampled daily
Orange bars fell below the 80% threshold.The three-day dip was not a regression. A promotion drove a spike in refund questions, and refunds have always been the weakest category — the agent did not change, the mix did.
Active alerts
Grade this answer
Humans grade clause by clause, never as one overall verdict. A single pass/fail button hides which rule was broken — and the rule is the thing you can actually fix.
Customer wrote
"I've been on hold for 45 minutes trying to reach someone about a charge I didn't authorise. Amazing service, truly."
Agent replied
Sounds like you've had quite the wait — sorry about that. I can look into the charge right now. Can you confirm the last four digits of the card and the date it appeared? Once I have those I'll flag it as disputed and you won't need to call anyone.
Rubric clauses
Grade each one independently."I'm not sure" is a first-class answer. Forcing a confident grade out of an unsure annotator is how rubrics stay broken — the uncertainty is the signal that the rule needs rewriting.
Try a change before you commit to it
Every edit here shows the cost of evaluating it. You always know what the next click spends before you spend it.
Preview on 3 sample cases
Run each 10 times — a single sample proves nothing.The tone fix worked and the retention flow did not move. Shipping now trades one problem for a clearer picture of the other — which is a reasonable call, as long as it is a call and not an accident.
Who decides what "correct" means
Model graders are cheap and consistent. Human graders are expensive and right. Verdict tracks how often each one agrees with the others, because a grader nobody checks is just an opinion at scale.
| Grader | Type | Cases graded | Agreement with humans | Cost / case |
|---|---|---|---|---|
| judge-sonnet-v3 | Model | 4,182 | 87.2% | $0.006 |
| judge-haiku-fastmiscalibrated | Model | 1,940 | 64.1% | $0.001 |
| M. Duarte | Human | 412 | 94.0% | — |
| A. Ferreira | Human | 388 | 91.3% | — |
| R. Okadadrifting | Human | 201 | 76.4% | — |
judge-haiku-fast costs a sixth of the price and agrees with humans two thirds of the time. That is not a cheaper grader, it is a different rubric wearing the same name — so it grades nothing that ships, only smoke tests. And the drift flag on a human grader matters just as much: annotators drift, and nobody notices unless it is measured.
Settings
The two settings that matter are at the bottom, and both exist to stop the product from lying to you.
General
Guardrails
On by default. Turning these off changes what the numbers mean.That last row is a setting that exists only to be permanently off — it answers the question every stakeholder asks, in the place they go looking for it, instead of leaving them to assume the feature was forgotten.