Key takeaways
- The rule that decides routing: a step whose output is checked by something else can run on a cheaper model. A step whose output nobody verifies cannot.
- Median tracked pricing in September 2026 was $0.95 per million input across 166 models, while frontier token prices rose 72% year over year. The gap between tiers is where routing pays.
- Of the 7 step types below, 4 are downgradeable and 3 are not. The 4 are usually the majority of calls in a run.
- Test before you commit: replay 100 real calls for one step against the cheaper model and grade with your existing acceptance check. That is an afternoon.
- Route per step, never per workload. A 10-step run rarely contains 10 decisions. It contains 1 or 2 decisions and 8 pieces of mechanical work.
Frontier token prices rose 72% over the twelve months to September 2026 while mid-tier prices fell 7.7%, according to BenchLM’s tracker of 166 models. For anyone running agents, that split between price tiers is not a market observation. It is an argument for doing something specific with your traces.
Most fleets pick one model for the whole agent. That decision is either overpaying on the trivial steps or underperforming on the hard ones, and usually both in the same run.
What is the rule for downgrading a step?
One question decides it: is this output checked by anything?
If a later step validates the result, a schema rejects it, or another component consumes it and would fail visibly on garbage, then a cheaper model producing a worse answer is a recoverable event. You will see it, and the run will either repair or fail loudly.
If nothing checks it, an error becomes the run’s premise. A misclassified intent, a wrong tool selection or a plan with a missing step propagates silently through everything after it, and the failure surfaces at the end as an inexplicably wrong answer with no obvious cause.
| Step type | Verified by | Route to |
|---|---|---|
| Extraction into a schema | The validator | Cheaper model |
| Classification into a fixed set | Set membership, downstream branch | Cheaper model |
| Formatting verified content | Format check | Cheaper model |
| Summarising already-checked material | The source remains available | Cheaper model |
| Deciding which tool to call | Nothing, until the run ends badly | Frontier |
| Deciding whether the task is complete | Nothing | Frontier |
| Producing output a person acts on | The person, too late | Frontier |
The pattern in the middle column is the whole method. Verification is what makes a cheap model safe, and its absence is what makes an expensive one necessary.
Where does the volume actually sit?
Usually in the four downgradeable step types (extraction, classification, formatting and summarising), which is why per-step model routing works.
A ten-step agent run rarely contains ten decisions. It contains one or two decisions and eight pieces of mechanical work: pull the fields out of this document, decide which of four categories this belongs to, turn this structure into prose, condense this result. Those steps are frequent, individually cheap to get wrong, and collectively the majority of the calls.
Count them in your own traces before assuming. Group spans by step name, count calls, and the distribution is usually more lopsided than anyone expects.
What actually degrades on a downgrade
Two things, consistently. Instruction adherence on complex tool schemas, and stability of structured output across runs.
Both are testable in an afternoon. Take a hundred real calls for one step from your trace store, replay them against the cheaper model, and grade with the acceptance check you already use, or with the smallest agent eval suite that gates a deploy if you have none. If the pass rate holds, route it. If it drops, you have learned something specific rather than assumed something general.
How do you implement this without a rewrite?
Per step, at the call site, with the model as a parameter rather than a constant.
| Step | What it gives you |
|---|---|
| 1. Name every step in your traces | The ability to count calls and tokens by step |
| 2. Make the model a per-step parameter | Routing without touching agent logic |
| 3. Replay 100 calls for 1 step on a cheaper model | Evidence instead of an argument |
| 4. Move 1 step, measure for 7 days | A real change with a reversible blast radius |
Step one is where most teams are blocked, and it is not a routing problem. If your spans do not carry a step name, you cannot count calls per step (the OpenTelemetry GenAI span attributes make that a query), cannot price them, and cannot tell afterwards whether the change helped.
What about routing between providers?
The same verification rule applies to routing between providers, with one extra constraint: the framework has to allow it.
Most agent frameworks now make provider routing a configuration change, including the vendor SDKs: the OpenAI Agents SDK accepts other providers through its LiteLLM extension, and the Claude Agent SDK can target Bedrock, Vertex or Azure. Check how your framework exposes the model per call before you design around it.
There is also a quieter cost. Two providers means two sets of tool-calling quirks, two structured-output behaviours and two failure modes to learn. That is real work, and it is only worth it if the volume on the downgraded step justifies it.
The question for your next cost review
Not “should we switch models”. Ask: how many calls did we make last week per step type, and how many of those steps have their output checked by something? The intersection of high volume and downstream verification is your entire savings opportunity, and it is visible in the traces you already have.
Frequently asked questions
Should you use a cheaper model for some agent steps?
Yes, for steps whose output is verified downstream. Extraction checked against a schema, classification into a fixed set, and formatting are all recoverable if wrong. Planning and tool selection are not, because an early error changes everything that follows.
Which agent steps need the most capable model?
The ones that make decisions the rest of the run depends on: deciding which tool to call, deciding whether a task is finished, and any step whose output is consumed directly by a person or by an irreversible action.
How much can model routing save on an agent workload?
It depends entirely on the share of calls that are downgradeable, which varies by workload. The right way to find out is to count calls by step type in your traces, then price each step separately, rather than to apply a published percentage.
What breaks when you downgrade a model for an agent step?
Usually instruction adherence on complex tool schemas, and stability of structured output. Both are testable before you commit: replay a hundred real calls for that step against the cheaper model and grade with your existing acceptance check.
Method and sources
- BenchLM, LLM Pricing Statistics, updated 18 September 2026: median $0.95 per million input and $3.88 per million output tokens across 166 models with live API pricing; frontier token prices up 72% and mid-tier prices down 7.7% over the preceding 12 months.
- OpenTelemetry Semantic Conventions for Generative AI, consulted for how per-step token attribution is captured.
- 2026 practitioner analyses describing hybrid architectures pairing a small model for high-volume simple tasks with a frontier API for hard reasoning.
- The routing rule proposed here is our editorial judgement. It is not a benchmark result and should be validated against your own acceptance checks.