Key takeaways
- A generic handler treats 8 distinct failure classes as one, because all of them arrive as an exception. Only 3 of the 8 resolve themselves on a second attempt.
- The rule that decides everything: retry when the cause is outside the model, repair when the cause is in the prompt, escalate when the cause is a permission or an ambiguity.
- A retried step costs more than the first attempt, because the failed attempt and its error are still in context. 3 attempts is not 3x the cost, it is more.
- Cap by class, not with one number: 2 attempts for transient infrastructure errors, 1 repair attempt for malformed output, 0 for permissions and ambiguity.
- Anything irreversible escalates on failure 1. A second attempt at a payment, a deletion or an external send is not a retry, it is a duplicate.
Every agent framework ships with a retry decorator and almost nobody configures it by failure type. The reason is structural: a network timeout and a malformed tool argument both surface as an exception, and a generic handler cannot tell them apart.
One of those failures will resolve itself if you wait. The other will reproduce exactly, at a higher price, for as many attempts as you allow.
What separates a retryable failure from a hopeless one?
Where the cause lives. That single test resolves almost every case.
If the cause is outside the model, meaning a network that dropped, a rate limit or an overloaded upstream, then nothing about your request was wrong and waiting is a reasonable strategy. If the cause is inside the model’s own output, such as a tool argument that does not validate or a plan referencing a step that does not exist, then re-running the same prompt against the same schema produces the same output. The retry is a payment for a repeat performance.
If the cause is a permission boundary or a genuine ambiguity in the instruction, no number of attempts helps, because the missing thing is authority or information that only a person can supply.
| Failure class | Cause lives | Action | Attempts |
|---|---|---|---|
| Network timeout | Outside | Retry with backoff | 2 |
| Rate limit, 429 | Outside | Retry with longer backoff | 2 |
| Upstream 5xx | Outside | Retry with backoff | 2 |
| Schema validation failure | In the output | Repair, then one attempt | 1 |
| Tool called with wrong arguments | In the output | Repair, then one attempt | 1 |
| Missing permission | Boundary | Escalate | 0 |
| Ambiguous instruction | The task | Escalate | 0 |
| Anything irreversible | Any | Escalate on first failure | 0 |
What does repair actually mean?
Not retrying harder. Changing the input before the next attempt.
A schema failure happens because the model’s understanding of the tool signature and the tool’s actual signature disagree. Sending the identical prompt again does not correct that disagreement. Sending the prompt plus the validator’s exact error message plus one correctly formed example often does, because you have added the missing information rather than repeated the request.
This is why repair belongs in a different column from retry. It has a different cost profile, a different success profile, and it needs the validator’s output to be captured rather than swallowed, which is a change most codebases need to make first.
The line that should never be crossed
In this decision table anything irreversible escalates on failure number 1. That covers money movement, deletion, sending a message to a third party, and publishing.
The reasoning is not about success rates. It is that the failure mode of a retry against a partially completed irreversible action is a duplicate, and duplicates in that category reach customers, regulators or both. If your tool cannot tell you whether the first attempt took effect, that is the bug to fix before you tune retry counts.
Why does a global retry limit not work?
Because it is simultaneously too generous and too strict.
Set it to three and every schema failure burns three attempts at growing context for a class of error that rarely resolves. Set it to one and you fail on transient timeouts that a second attempt would have cleared. There is no single number that is correct for both, which is the whole argument for classifying.
A single limit also compounds where the stack is layered. Google’s Site Reliability Engineering, chapter 22, works the arithmetic for plain services: three layers each retrying three times turn one user action into 64 attempts against the database at the bottom. An agent stack layers the same way, with the model call, the tool client and the HTTP client each carrying their own default.
There is a cost dimension too. A retried step is larger than the original, because the failed attempt and its error are usually still in the context when the retry runs. Three attempts is not three times the cost. It is more, and it grows with each one, which is the same accumulation that makes the cost of a single agent task hard to predict before you run it.
What does each class cost you to get wrong?
The two directions of error in the eight-class table above are not symmetrical, which is why one global limit cannot balance them. Over-retrying and over-escalating fail in different currencies: one spends tokens, the other spends a person’s attention.
| Failure class | Retrying when you should not | Escalating when you should not | Recommended limit |
|---|---|---|---|
| Transient infrastructure | Nothing, it usually succeeds | A human handles a 200ms blip | 2 |
| Malformed output | Full context re-sent, low success | A human fixes a comma | 1 repair |
| Missing permission | Unbounded attempts, 0 succeed | Nothing, this is correct | 0 |
| Ambiguous instruction | Confident wrong answers | Nothing, this is correct | 0 |
| Irreversible action | Duplicate payment or send | A human confirms once | 0 |
The bottom row is why the policy is asymmetric. The cost of over-escalating is a person spending 30 seconds. The cost of over-retrying is a customer charged twice.
What do you need in place to implement a per-class retry policy?
Three things, in order, before the eight-class table above is worth anything in code.
- The failure class has to reach the handler. Most codebases catch a generic exception and lose the distinction at the first
except. Preserve the error type and the validator output. - An escalation path that is not an exception. Escalation means a queue item a person can act on, with enough context to act. A run that raises and dies has not escalated, it has given up.
- Per-class counters in your telemetry. The OpenTelemetry GenAI conventions already put token usage on the span, so attempt counts and cost land in the same query once the failure class is an attribute. Without that split you cannot tell whether the policy is working, and tuning becomes guesswork.
The question for your next reliability review
Not “how many times do we retry”. Ask: what share of last month’s retried attempts succeeded, split by failure class? In most fleets one class carries nearly all the successes and the rest is money spent on repetition.
Frequently asked questions
Should an AI agent retry a failed tool call?
Only when the cause is external and transient: timeouts, rate limits, and 5xx responses. Failures originating in the model's own output, such as a malformed tool argument, reproduce on retry because the same prompt against the same schema produces the same mismatch.
What is the difference between retrying and repairing an agent failure?
A retry re-sends the same request unchanged. A repair injects new information first, typically the validator's error message and a correctly formed example, then attempts once more. Retrying a schema failure is usually waste; repairing it often works.
How many times should an agent retry?
By class rather than by a single number. Two attempts with backoff for transient infrastructure errors, one repair attempt for malformed output, and zero for permission errors or ambiguous instructions, which should go to a human immediately.
Why do retries make agent costs rise so quickly?
Because a retried step re-sends the accumulated context, and the failed attempt plus its error message is usually still in that context. The second attempt is therefore larger than the first, and a third larger again.
Method and sources
- Site Reliability Engineering, chapter 22, Addressing Cascading Failures, Google, consulted for retry amplification across layers, retry budgets and randomised exponential backoff. Source of the figure that three layers retrying three times each produce 64 attempts at the bottom.
- Token Economics for LLM Agents: A Dual-View Study from Computing and Economics, arXiv:2605.09104, consulted on per-attempt cost accumulation.
- OpenTelemetry Semantic Conventions for Generative AI, which moved to a dedicated repository and remains in development, consulted for how retried attempts appear in a span tree.
- We found no primary published study measuring per-class retry success rates for LLM agents, so no such rate appears in this article. The repair-then-retry description is a restatement of how validator feedback loops work, not a measured result.
- The classification and limits recommended in this article are our editorial judgement. They are not measured success rates and should be tuned against your own telemetry.