Key takeaways
- Evaluation gaps are reported as the top blocker to production by 64% of enterprise leaders, ahead of governance and model reliability.
- The minimum viable suite is 4 components and about 5 working days: a held-out task set, a deterministic grader, a pass-rate threshold that blocks a deploy, and an intake path for new failures.
- Start with fifty real tasks, not five hundred synthetic ones. A small set drawn from production predicts better than a large set someone invented.
- 5 of the 6 checks worth running need no model at all: schema validity, required fields, citation existence, numeric match and outcome in the target system.
- A suite that has caught 0 new failures in 30 days has stopped measuring capability and started measuring memory.
Sixty-four percent of enterprise leaders name evaluation gaps as the thing stopping their agents from reaching production, the figure most widely quoted from 2026 Forrester and Anaconda panel data. It outranks governance friction at 57% and model reliability at 51%.
That is a strange result, because building the first useful version of an eval suite takes about a week. The reason it does not get built is that it produces nothing to demo, and the reason it eventually gets built is that something went wrong in front of a customer.
What is the smallest suite that is worth having?
Four components. Anything less does not gate a deploy, which means it is documentation rather than a control.
| Component | Recommended size | Effort, days | Replaces |
|---|---|---|---|
| Held-out set of real tasks | 50 tasks, 40 from failures | 2 to 3 | Someone spot-checking outputs |
| Deterministic grader | 5 mechanical checks | 1 | Reading JSON by eye |
| Pass-rate threshold that blocks a deploy | 1 number, set once | 0.1 | Hoping the change was fine |
| Intake path for new failures | 10 minutes per incident | 0.5 | Failures noticed and forgotten |
| Total | 4 components | ~5 days |
The fourth is the one that decides whether the suite is alive in six months. Without a path from production failure to test case, the set freezes on the day it was written and slowly stops describing your system.
Where do the fifty tasks come from?
Production, and specifically from the runs that went wrong.
The instinct is to write test cases that cover the feature surface evenly. That produces a set dominated by cases the agent already handles, which is why so many suites report 95% and predict nothing. A set weighted toward observed failures is smaller, less tidy, and tells you something, which is most of the distance between a pilot that passes and a deployment that holds.
Practically: pull the last month of runs, filter to the ones that escalated, errored, or were retried more than once, and read forty of them. Most will cluster into five or six recognisable failure modes. Take a handful from each cluster, add ten straightforward cases so regressions on the easy path are visible, and you have your set.
Why fifty and not five hundred
Because you have to define the correct outcome for each one by hand, and that is the expensive part. Fifty cases you have actually adjudicated beat five hundred whose expected outputs were generated by a model and never checked. The second set measures agreement with the generator, which is not a property you care about.
Size is also not what makes a score portable. Same Model, Different Harness (arXiv:2608.26218) froze model weights on a 169-task SWE-bench Verified cohort and moved mean per-task fail-to-pass from 28% to 49% by changing the harness alone, which is why a benchmark number travels badly between setups. Your suite measures your agent inside your harness, and that is the only configuration you ship.
What should the grader check?
Everything mechanical first, and only then anything subjective.
| Check | Method | Use a model? |
|---|---|---|
| Output parses against the schema | Validator | No |
| Required fields present and typed | Validator | No |
| Cited document exists and was retrieved | Lookup | No |
| Numeric answer matches expected | Comparison with tolerance | No |
| Task outcome achieved in the target system | Query the system | No |
| Explanation is clear and complete | Model grading, validated first | Yes, with caution |
| Total needing a model | 1 of 6 | 17% |
Five of those six rows need no model at all, and in most agent workloads they cover the majority of real failures. Teams reach for a model grader early because it feels more sophisticated, then inherit position bias, verbosity bias and a grader nobody has validated.
The size of that inheritance is measured. Reliability without Validity (arXiv:2606.19544) ran 21 judges across nine providers over roughly 541,000 judgments and found exact-match agreement overstating judge quality by 33 to 41 percentage points against Cohen’s kappa. Two production judges held test-retest reliability above 0.95 while carrying position bias above 0.10, so a grader can be perfectly repeatable and still wrong in the same direction every time.
If you do use one, validate it before you trust it: reverse the order of candidates and count how many grades flip. That single test disqualifies more grading pipelines than any other, and judges that prefer whichever answer came first are the common case rather than the exception.
How do you keep it from going stale?
By treating every production failure as an unpaid contribution to the test set.
The rule that works is mechanical: when a run fails in production and someone investigates, the last step of the investigation is adding that case to the suite with its correct outcome. Not a similar case, that one. It takes ten minutes and it is the only mechanism that keeps the set describing the system you actually have.
Run count matters as much as case count. Beyond Accuracy (arXiv:2511.14136) reports enterprise agent success falling from 60% on a single run to 25% when the same task has to succeed 8 runs in a row, and its CLEAR framework correlating with production success at 0.83 against 0.41 for accuracy-only scoring. A suite that runs each case once is reporting a best case.
The health check is simple. If the suite has not caught a new failure in a month, it is no longer measuring capability. It is measuring whether you broke something you already fixed, which is worth having and is not the same job.
The question to bring to your next planning meeting
Not “should we build evals”. Ask: if someone shipped a prompt change today that made us 10% worse, which check would fail and who would see it? If the honest answer is a customer, the suite is the next thing to build, ahead of any feature.
Frequently asked questions
How do you evaluate an AI agent?
Assemble a held-out set of real tasks with known-good outcomes, define an acceptance check per task, run every candidate change against the whole set, and compare pass rates. For agents specifically, grade the final outcome rather than intermediate reasoning, because there are many valid paths to a correct result.
How many test cases does an agent eval suite need?
Start at around fifty drawn from real production traffic, weighted toward cases that have failed. Statistical precision matters less at this stage than coverage of the failure modes you actually see. Grow the set by adding each new production failure rather than by generating variants.
Should you use an LLM to grade agent outputs?
Only for what cannot be checked mechanically. Schema validity, required fields, citation existence and format are deterministic checks and should stay that way. Reserve model grading for genuinely subjective quality, and validate the grader against known answers before trusting it.
What pass rate should gate a deployment?
Whatever your current production rate is, minus a tolerance you set deliberately. The absolute number matters far less than the rule that a change which lowers it does not ship without someone accepting the trade in writing.
Method and sources
- Top-blocker figures for agent production deployment, evaluation gaps at 64% of leaders, governance friction at 57% and model reliability at 51%, are credited across 2026 coverage to Forrester and Anaconda enterprise panel data and are quoted here as reported. We did not find a primary page carrying all three figures together, so treat the breakdown as directional.
- Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Bias, arXiv:2606.19544, consulted on the limits of model-based grading.
- Same Model, Different Harness: Different Coding-Agent Results, arXiv:2608.26218, reporting that on a 169-task SWE-bench Verified cohort a harness change alone moved mean per-task fail-to-pass from 28% to 49% with model weights frozen.
- Beyond Accuracy: A Multi-Dimensional Framework for Evaluating Enterprise Agentic AI Systems, arXiv:2511.14136, reporting agent success falling from 60% on a single run to 25% across 8 consecutive runs, and its CLEAR framework correlating with production success at 0.83 against 0.41 for accuracy-only evaluation.
- The build order and thresholds recommended in this article are our editorial judgement, not findings from a study.