Key takeaways
- A success rate without a definition is not a number. The same run can be scored as success or failure depending on whether escalation counts, which is why two teams both reporting 90% are usually not comparable.
- Report pass@1 and pass^k together. τ-bench found agents solving under 50% of retail tasks on one attempt falling below 25% at pass^8, the share succeeding on all 8 identical tries.
- Cost belongs per completed task, not per call or per token. A model that halves token cost and doubles the escalation rate improves every per-call metric and makes the system worse.
- Split tool failures into 2 counters: schema failures and semantic failures. The Structured Output Benchmark found near-perfect schema compliance alongside a best value accuracy of 83.0% on text, so a single error rate hides the larger half.
- Our threshold recommendations below are editorial judgement, not measured baselines. Tune them against your own telemetry before alerting on any of them.
Two teams report 90% task success. One counts an escalation to a human as a failure, the other counts it as a success because the customer was served. The numbers are not comparable and nothing on either dashboard says so.
Agent metrics fail in the definition, not in the collection. What follows is the set we would keep, with the definition that makes each one mean the same thing twice.
What makes a metric comparable between two teams?
Three properties, and most agent metrics have one.
A comparable metric names its denominator, states its unit of work, and says what it does with escalations. Task success rate has none of those by default. Is the denominator tasks attempted or tasks admitted? Is the unit of work a user request or an orchestrator run, which may be several per request? And does a run that stopped and asked a person count in the numerator, the denominator, or neither?
Those three answers move a reported rate by tens of points on the same data. Publishing them alongside the number costs one line and is the difference between a metric and a mood.
Which twelve metrics are worth keeping?
| Metric | Family | Definition | Unit | Review threshold we would set |
|---|---|---|---|---|
| Task success rate, pass@1 | Outcome | Tasks meeting the acceptance check on the first attempt, over tasks admitted | % | Report always |
| pass^k | Outcome | Tasks succeeding on all k identical attempts, over tasks admitted | % | k = 8 |
| Escalation rate | Outcome | Runs stopped for a person, over runs admitted | % | Alert above 20% |
| Abandonment rate | Outcome | Runs that ended without success or escalation | % | Alert above 1% |
| Cost per completed task | Cost | Total spend in a window, over tasks that succeeded | currency | Weekly |
| Tokens per completed task | Cost | Input plus output tokens, over tasks that succeeded | tokens | Weekly |
| Non-token spend share | Cost | Per-search, runtime and container charges, over total spend | % | Alert above 25% |
| Time to first token | Latency | Request sent to first streamed token | ms, p95 | Per model |
| Task wall-clock | Latency | Admission to terminal state, including waits for people | s, p95 | Per task class |
| Schema failure rate | Reliability | Tool calls rejected by the schema, over tool calls | % | Alert above 2% |
| Semantic failure rate | Reliability | Tool calls passing the schema and failing a check it cannot express | % | Alert above 5% |
| Retry success rate by class | Reliability | Retried attempts that succeeded, split by failure class | % | Monthly, per class |
The thresholds in the last column are starting points we would defend in a design review and would expect to move within a quarter of real telemetry. Treat them as shapes rather than targets.
Why does the outcome family need two success metrics?
Because one attempt tells you what the agent can do and repeated attempts tell you what it will do.
τ-bench introduced pass^k for exactly this: the share of tasks that succeed on all k independent trials, rather than the share where at least one trial succeeded. The two diverge fast. In the retail domain, agents solving under half of tasks on a single attempt fall below 25% at pass^8.
That gap is the number a pilot never shows and production never stops showing. A demo runs each task once. A fleet runs the same task class thousands of times and experiences the distribution, which is most of what separates a working pilot from a working deployment.
Abandonment deserves its own row for a related reason. A run that ends in neither success nor escalation is invisible in both of the first two metrics, and it is the single most expensive outcome: money spent, nothing delivered, nobody told.
What should cost be measured per?
Per completed task, and nothing else belongs on the headline.
Per-call and per-token costs both improve under a change that makes the system worse. Switch to a cheaper model that needs three attempts where the previous one needed one, and cost per call falls while cost per completed task rises. The metric that survives that substitution is the one with completed work in its denominator, which is the same arithmetic that a per-task cost calculation has to carry through.
The non-token share is the row teams add last and regret not having. Search fees, container hours and session runtime do not appear in any token counter, so a pipeline that only sums gen_ai.usage.input_tokens and gen_ai.usage.output_tokens reports a cost that is confidently wrong by whatever that share happens to be.
Which reliability metrics catch what outcome metrics hide?
The ones that split failures by cause rather than counting them.
A single tool error rate merges two populations with nothing in common. The Structured Output Benchmark measured 21 models and found near-perfect schema compliance alongside a best value accuracy of 83.0% on text inputs, which means that even for the best of them roughly one in six text extractions is structurally perfect and factually wrong. Those two failures need different handling: one is a repair candidate with a validator message to inject, the other has nothing to repair with.
Retry success rate by class is the metric that makes a retry policy tunable at all. Google’s Site Reliability Engineering, chapter 22, argues for a retry budget enforced where calls are issued, and a budget needs to know which classes are earning their attempts. Without the split you cannot tell whether your retries are recovering transient timeouts or paying repeatedly for a failure that reproduces every time.
Where should these numbers come from?
From the trace, on the same spans, so that a per-task join is possible at all.
The OpenTelemetry GenAI semantic conventions define the attributes this set needs: gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for the cost family, gen_ai.usage.cache_read.input_tokens to separate cached from fresh input, gen_ai.tool.name for the reliability split, gen_ai.agent.version to keep comparisons honest across deploys, and gen_ai.conversation.id to group the spans of one task. The conventions have moved to a dedicated repository and remain in development, so pin the version you instrument against.
The practical consequence is that these twelve are one query rather than four dashboards, which is what putting token counts on the span buys you.
| Family | Metrics in it | What it hides on its own | Companion that exposes it |
|---|---|---|---|
| Outcome | 4 | Variance across repeats, and runs that ended in nothing | pass^k at k = 8, abandonment rate |
| Cost | 3 | Charges not billed in tokens, and work paid for but not delivered | Non-token spend share, cost per completed task |
| Latency | 2 | Time a run spent waiting for a person rather than a model | Task wall-clock at p95 against time to first token |
| Reliability | 3 | Wrong values inside structurally valid tool calls | Semantic failure rate against schema failure rate |
What should you not measure?
Three things that look like metrics and are not.
- Average tokens per call. It moves when prompts change, tool schemas change or caching changes, and it tells you nothing about whether work got done. Tokens per completed task carries the same information with a denominator that means something.
- Number of tool calls per run. Lower is not better. An agent that makes 4 calls and gets the answer beats one that makes 2 and guesses.
- A single blended quality score. Blending outcome, cost and latency into one figure destroys the only property that makes the set useful, which is that the four families move independently and the trade-offs between them are the decision.
The question to take into the next review is not which metrics you have. Ask: for the same week’s data, can you state the denominator and the escalation treatment behind every percentage on the dashboard? Whichever number fails that test is the one people have been arguing about.
Frequently asked questions
What metrics should you track for an AI agent in production?
Twelve, across four families: outcome, cost, latency and reliability. Outcome needs both pass@1 and pass^k, cost needs to be per completed task, and tool failures need splitting into schema failures and semantic failures.
What is pass^k and how is it different from pass@k?
pass^k is the share of tasks that succeed on all k identical attempts, introduced in τ-bench to measure reliability. pass@k is the share where at least one of k attempts succeeds, which rewards variance rather than penalising it.
Why measure cost per completed task instead of per API call?
Because a task that fails and is retried consumed real money and produced nothing. Per-call cost improves whenever calls get cheaper, even if the number of calls needed to finish a task rises faster.
Where should agent metrics come from?
From the trace, using the OpenTelemetry GenAI conventions, so that token counts, tool names, agent versions and outcomes sit on the same spans. Metrics assembled from separate systems cannot be joined per task.
Method and sources
- τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains, Yao, Shinn, Razavi and Narasimhan, arXiv:2406.12045. Source of the pass^k definition as published, of the under-50% single-attempt result, and of pass^8 below 25% in the retail domain.
- OpenTelemetry Semantic Conventions for Generative AI, consulted for the attribute names cited here, including gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens, gen_ai.operation.name, gen_ai.agent.version, gen_ai.tool.name and gen_ai.conversation.id. The conventions moved to a dedicated repository and remain in development.
- The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models, arXiv:2604.25359. Source of the best value accuracy of 83.0% on text against near-perfect schema compliance, across 21 models.
- Site Reliability Engineering, chapter 22, Addressing Cascading Failures, Google, consulted for the retry budget pattern that the retry metrics here are built to support.
- The twelve-metric set, the groupings and every threshold in the last column are our editorial judgement. They are not measured baselines and we publish no success rates, costs or latencies of our own.