Key takeaways
- On 94 tasks drawn from 7 companies, the best of 14 model-scaffold configurations scored 64.41 out of 100 in the AlphaEval benchmark. A suite built from curated tasks cannot tell you where you sit on that scale.
- The agent product around the model moved scores as much as the model did: Opus 4.6 scored 64.41 in Claude Code and 53.45 in Codex, an 11-point spread with the model held fixed.
- Domains differed widely: the average was 62.0 in Technology Research and 30.0 in Human Resources, so one aggregate number hides which workflows are weak.
- A single run overstates reliability. In τ-bench's retail domain, agents solving under 50% of tasks on one attempt fell below 25% when all 8 repeats had to succeed.
- Before you tell anyone what the suite did not measure, list 3 gaps to close first: task provenance, the scaffold you actually deploy, and repeated runs of the same task.
AlphaEval, a benchmark posted to arXiv on 14 April 2026, took 94 tasks from 7 companies’ real deployments. The best of 14 model-and-agent configurations scored 64.41 out of 100, and the weakest domain averaged 30.0.
Your suite is green and the deployment is not. The question for tomorrow’s meeting is not whether the tests were wrong but which conditions they never reproduced.
What did the production-sourced benchmark actually measure?
It measured agents on work that paying customers had asked for, not on tasks written to be gradable.
AlphaEval (Lu et al., arXiv:2604.12162) contains 94 tasks across 6 occupational domains, sourced from 7 companies. The authors describe the contrast with existing benchmarks as retrospectively curated tasks with well-specified requirements, against production requirements that carry implicit constraints. They evaluated complete agent products as the commercial systems they are, not bare models: six models and four products, 14 configurations in all.
The best configuration scored 64.41. That figure is not a failure rate for agents in general. It is the ceiling on one set of tasks that no one built to be easy, and it shows how far a clean benchmark score can sit from a score on real work.
How far apart were the domains?
Far enough that a single number hides the problem. The average was 62.0 in Technology Research and 30.0 in Human Resources. A team whose workload looks like the second domain should not take comfort from the first.
Does the agent product around the model change the result?
Yes, by as much as swapping the model did in several comparisons.
| Model | Agent product | Score out of 100 | Spread for the model |
|---|---|---|---|
| Claude Opus 4.6 | Claude Code | 64.41 | 10.96 points |
| Claude Opus 4.6 | Codex | 53.45 | 10.96 points |
| GPT-5.2 | GitHub Copilot | 54.91 | 15.44 points |
| GPT-5.2 | Claude Code | 39.47 | 15.44 points |
AlphaEval’s authors conclude that scaffold choice matters as much as model choice. The pairs are models available in April 2026, so read them as evidence about scaffolds, not as a ranking of current models. For a team, the consequence is concrete: an evaluation run through a test harness is a measurement of the harness plus the model, and the harness you test is rarely the one you deploy. The suite you build first should therefore run through your production agent loop, not a convenient stand-in.
Why does one successful run overstate reliability?
Because production repeats the same task class thousands of times and meets the whole distribution.
τ-bench measured this directly with pass^k, the share of tasks that succeed on all k repeats. In its retail domain, agents that solved under 50% of tasks on one attempt fell below 25% at pass^8. A suite that runs each case once reports the first number and production lives with the second. This is the same gap that separates a working pilot from a working deployment, seen from the test side rather than the rollout side.
Tool arguments add a second hidden layer. The Structured Output Benchmark found near-perfect schema compliance across 21 models and a best value accuracy of 83.0% on text. A test that checks only that a call parsed will pass calls whose values are wrong, which is the split between schema and value accuracy.
What conditions does a typical suite skip?
Most of the ones below. This comparison is our editorial judgement, with the evidence it rests on in the right-hand column.
| Condition | Typical suite | Production | Evidence in this text |
|---|---|---|---|
| Task origin | Written for gradability, clear requirements | Implicit constraints, mixed inputs | AlphaEval: best score 64.41 of 100 |
| Agent loop | Harness chosen for convenience | The scaffold you ship | AlphaEval: 11 and 15 point spreads |
| Repeats | Each case once | Same task class run constantly | τ-bench: under 50% falls below 25% at pass^8 |
| Tool arguments | Call parses and runs | Values must be right | Structured Output Benchmark: 83.0% best value accuracy |
Contamination is a fifth condition that corrupts public benchmarks specifically, so it matters when your suite borrows from them.
What should you close first?
Close three gaps, in this order, and say so in the status you send up.
- Add real tasks. Sample a set of recent production tasks, score them the way you score the suite, and compare. If the suite’s own cases score far higher, you now have a number to give the person asking.
- Evaluate the shipped loop. Run the suite through the exact scaffold, tool wiring and retry policy in production. AlphaEval’s 11 and 15 point scaffold spreads are the size of error to expect from skipping this.
- Repeat each case. Report pass@1 next to pass^k, with k around 8 as in τ-bench, so the reliability number and the capability number sit side by side.
The question to answer before the meeting is: which of these three gaps does our suite have, and what did the last production incident fall inside? Whichever it is, that is what the tests did not measure.
Frequently asked questions
Why does my AI agent pass tests but fail in production?
Because the tests usually measure tasks you wrote in advance, with clear requirements, run once, in a harness that differs from what you deploy. Production supplies vague requirements, heterogeneous inputs and repeated runs of the same task class.
How do I know whether my eval suite predicts production results?
Compare it against a sample of real production tasks scored the same way. If the suite scores much higher on its own tasks than on that sample, it measures the suite and not your workload.
Does the agent framework matter as much as the model?
On AlphaEval it did. Claude Opus 4.6 scored 64.41 in Claude Code and 53.45 in Codex, and GPT-5.2 scored 39.47 in Claude Code against 54.91 in GitHub Copilot, so evaluate the scaffold you ship.
How many times should I run each eval task?
More than once. τ-bench reports pass^k, the share of tasks succeeding on all k repeats, because a single success hides tasks the agent solves only some of the time.
Method and sources
- AlphaEval: Evaluating Agents in Production, Lu et al., arXiv:2604.12162, April 2026. Source of the 94 tasks, 7 companies, 6 occupational domains, 14 configurations, the 64.41 best score, the 62.0 and 30.0 domain averages and the four scaffold comparisons. Preprint; scores are the authors' and we did not rerun them.
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, Yao, Shinn, Razavi and Narasimhan, arXiv:2406.12045. Source of the pass^k definition, the under-50% single-attempt result and the below-25% result at pass^8 in the retail domain.
- The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models, arXiv:2604.25359. Source of the best value accuracy of 83.0% on text across 21 models, against near-perfect schema compliance.
- The three-gap checklist and the comparison of suite and production conditions are our editorial judgement. We ran no experiment of our own and do not claim that any specific agent in the AlphaEval tables passed any specific test suite.