Key takeaways
- Prompt caching matches on a prefix. Change one token near the start and everything after it is recomputed at full price, however stable the rest was.
- Agents break prefixes for 5 boring reasons, and 4 of the 5 sit in the first 100 tokens: a timestamp, unsorted tool definitions, retrieved documents placed too early, or an A/B flag.
- A February 2026 evaluation of caching across OpenAI, Anthropic and Google Vertex AI found cache invalidation in multi-turn agent interactions to be the central failure, not cache capacity.
- Order context by volatility across 6 positions, most stable first. Sorting tool definitions is 1 line of code and the highest-leverage line in this article.
- Reported savings of 50% to 90% on input cost are real and routinely unrealised, because the saving applies only to the part of the prompt that did not move.
Does ChatGPT recommend your competitor instead of you?
Check how AI assistants describe your company, and whose name they give when someone asks for a recommendation in your category. Run an audit today, from $19.
Prompt caching is the largest single lever on an agent bill, and the reason most teams see a fraction of the advertised saving is mechanical rather than mysterious. Caching matches on a prefix. Change one token near the front and every token after it is recomputed at full price.
A February 2026 evaluation of caching across three providers reached the same conclusion from the research side: for long-horizon agent tasks, the problem is not cache capacity. It is invalidation.
What actually invalidates the cache?
Anything that moves earlier in the prompt than the thing you wanted to keep.
| What changes | Where it usually sits | Cost of the mistake |
|---|---|---|
| Timestamp or session id in the system prompt | First 50 tokens | Entire prompt recomputed, every turn |
| Tool definitions serialised in map order | Immediately after system prompt | Cache hits become random |
| Retrieved documents before instructions | Front of prompt | Nothing after retrieval is ever reused |
| History prepended rather than appended | Grows at the front | Cache resets on every turn |
| A/B flag injected into the preamble | First 100 tokens | Half your traffic never hits cache |
None of these is a caching problem. All of them are prompt assembly problems that only become visible when you look at a cost line and cannot explain it.
The tool-definition trap
The second row deserves attention because it is the least obvious. If your tool definitions are built by iterating a dictionary, their serialised order can differ between processes even when the set is identical. The prompt is semantically the same on every call and byte-different often enough to destroy the hit rate.
Sorting tool definitions by name costs one line and is the highest-leverage line in this article.
How should context be ordered?
By volatility, most stable first. Most agent implementations do the reverse, because they were written in the order a human thinks about the problem rather than the order the cache reads it.
| Position | Content | Changes how often |
|---|---|---|
| 1 | System prompt, with no dynamic values | Per deploy |
| 2 | Tool definitions, sorted deterministically | Per deploy |
| 3 | Stable reference material | Per session |
| 4 | Retrieved documents for this task | Per task |
| 5 | Conversation history, appended | Per turn |
| 6 | The current turn | Every call |
If today’s date genuinely must be in context, it belongs at position five, not inside the system prompt. Moving it is usually a ten-minute change that converts a permanently cold cache into a warm one.
What the ordering is worth, in tokens
| Prompt arrangement | Stable prefix | Cacheable share | Input tokens billed over 10 turns |
|---|---|---|---|
| Timestamp in system prompt | 0 tokens | 0% | 100,000 |
| Retrieved docs before instructions | 2,000 | 20% | 82,000 |
| Volatility ordering applied | 8,000 | 80% | 28,000 |
Why does the bill still go up?
Because a cheaper unit invites more units, and agents are unusually good at consuming whatever headroom you give them.
Once caching lands, longer chains become affordable, larger retrieved context becomes affordable, and running three candidate plans instead of one becomes affordable. Each decision is locally reasonable. The invoice does not care that the per-token price fell.
This is the same pattern the wider inference market showed this year: per-token prices dropped sharply while plenty of teams paid more. Caching improves unit economics. It does not impose a budget, and nothing except a budget imposes a budget.
What should you check this week?
Three things, in order of how quickly they pay back.
- Diff two consecutive prompts, byte for byte. Not the rendered version, the exact string sent. The first difference is where your cache stops. Teams are routinely surprised by how early it appears.
- Sort your tool definitions. One line, deterministic order, measurable the same day.
- Look at cached versus uncached input tokens per completed task. If your provider reports both and the cached share is under half on a repetitive workload, the ordering above is your fix.
The question for your next cost review
Not “are we using prompt caching”. Everyone is, nominally. Ask: what percentage of input tokens were served from cache last week, and where does the first byte-difference between two consecutive calls occur? The second half of that question is where the money is.
Frequently asked questions
How does prompt caching work?
The provider stores the computed state for a prefix of your prompt and reuses it when the next request begins with exactly the same tokens. Matching is on the prefix, so reuse stops at the first byte that differs. Everything from that point on is processed at the normal rate.
Why does prompt caching not work for my agent?
Almost always because something at the front of the prompt changes between turns. Common causes are a timestamp or session identifier in the system prompt, tool definitions serialised in a different order, retrieved documents placed before the stable instructions, or history prepended instead of appended.
How much can prompt caching save on an agent workload?
Reported figures for input cost run from 50% to 90% depending on provider and on how much of the prompt is genuinely stable. Treat those as an upper bound. The realised saving is proportional to the share of your prompt that is identical, in the same order, on the next call.
Does prompt caching reduce the total AI bill?
Not reliably. Per-token cost falls, and teams respond by running longer chains and larger context because they can now afford to. The invoice frequently rises anyway. Caching improves unit economics; it does not by itself impose a budget.
Method and sources
- Elias Lumer, Faheem Nizar, Akshaya Jangiti, Kevin Frank, Anmol Gulati, Mandar Phadate and Vamse Kumar Subbiah, Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks, arXiv:2601.06007v2, 3 February 2026. Primary source for the finding that cache invalidation across multi-turn agent interactions is the central problem, evaluated across OpenAI, Anthropic and Google Vertex AI context caching.
- 2026 practitioner write-ups reporting input cost reductions of 50% to 90% and typical agent workload savings of 40% to 70%. Cited as reported; we have not reproduced these measurements.
- Provider documentation for prompt caching from OpenAI, Anthropic and Google Vertex AI, consulted for the prefix-matching behaviour described in this article.
- The observation that teams increase usage after caching lowers unit cost appears in several 2026 cost analyses and matches induced demand behaviour reported elsewhere in inference spending.
- No figures in this article are our own measurements.