Measuring whether an agent actually works: datasets, graders, regression suites, production traces.
Why an AI agent that passed its tests fails in production, using AlphaEval's 94 tasks from 7 companies, where the best score was 64.41 out of 100.
Two teams reporting 90% success usually measure different things. Here are the twelve agent metrics worth keeping, each with a definition that fixes it.
Evaluation gaps are the most-cited reason agent pilots never ship. Here is the smallest suite that actually gates a deploy, and the order to build it in.
Surveys put agent pilot failure at 86 to 89 percent, and the most-cited blocker is not the model. It is evaluation infrastructure nobody budgeted for.
SWE-bench Verified leaders cluster inside one point, while the same weights swing 10 to 20 points across harnesses. What a leaderboard cannot tell you.
Automated graders agree with each other beautifully. A 2026 RAND study found that says nothing about whether they agree with reality.