Key takeaways
- Published evaluations report accuracy declining as input length grows across every major model tested, including Claude Sonnet 4, GPT-4.1, Qwen3-32B and Gemini 2.5 Flash.
- Larger context windows move the point where degradation begins. They do not remove the degradation, which is why a million tokens is a capacity figure and not a quality guarantee.
- Degradation accelerates as the semantic distance between the query and the target information grows. Needle-in-a-haystack tests use near-verbatim matches and therefore understate the real effect.
- Agent context is multi-turn and accumulated. A 20-step run can carry 10x the context of step 1, all of it inside the window and all of it competing for attention.
- Treat context as a budget with an eviction policy. The 4 moves below remove different things, and capping run length removes the tail of the curve entirely.
Does ChatGPT recommend your competitor instead of you?
Check how AI assistants describe your company, and whose name they give when someone asks for a recommendation in your category. Run an audit today, from $19.
Published evaluations of long-context behaviour report the same result across every major model tested: accuracy falls as the input gets longer, whether or not the answer is in there.
Agents accumulate context as a matter of design. Every tool result, every intermediate step, every retrieved document goes into the window and stays. Which means the default trajectory of a long agent run is along the part of the curve that gets worse.
What does the research actually show?
Not truncation. Something less convenient than truncation.
Truncation is easy to reason about: the information fell out of the window, so the model could not use it. Context rot is the case where the information is present, well within the window, and the model still fails to use it. Reported evaluations show this across Claude Sonnet 4, GPT-4.1, Qwen3-32B and Gemini 2.5 Flash, which is a wide enough spread to treat as a property of the approach rather than of one lab.
| Reported finding | Implication for an agent run |
|---|---|
| Accuracy declines with input length across all models tested | Long runs degrade even when nothing is dropped |
| Million-token windows move the onset, not the effect | Window size is headroom, not a quality guarantee |
| Degradation accelerates as query and target diverge semantically | Paraphrased or implied targets fail earlier than benchmarks suggest |
| Benchmarks are single-turn; agent context is accumulated | The published numbers are the optimistic case |
The third row is the one that changes how you read vendor claims. A needle-in-a-haystack score is measured with a target that closely matches the query. Real agent tasks ask about things stated differently, three steps ago, by a tool.
How fast context actually accumulates
| Step | Context at start of step | Multiple of step 1 |
|---|---|---|
| 1 | 2,000 | 1x |
| 5 | 5,200 | 2.6x |
| 10 | 9,200 | 4.6x |
| 20 | 17,200 | 8.6x |
Why does this hit agents harder than chat?
Because of what ends up in the window and who decides.
In a chat product, context grows with the conversation and a person is implicitly curating it. In an agent run, context grows with every tool call, and nothing curates it. A search that returned twelve results puts twelve results in the window. A file read puts the file in the window. None of it leaves.
By step twenty you have a context that is mostly the wreckage of steps one through nineteen, and the model is being asked to find the relevant part of it without help.
The monitoring case is worse
There is a specific version of this worth naming, because it looks like a safety control and behaves like a blind spot. If you run a second model over a long agent transcript to check for dangerous actions, that monitor is itself a long-context task. Research on classifier context rot reports the monitor degrading with transcript length, which means your oversight gets least reliable exactly when the run got longest and most complicated.
What should you actually change?
Treat context as a budget with an eviction policy. Four moves, in order of payoff.
| Move | What it removes | Cost |
|---|---|---|
| Summarise completed steps | Full transcripts of work already finished | One extra model call per compaction |
| Keep tool output out of context by default | Raw payloads nobody reads again | A later step may need to re-fetch |
| Re-retrieve on demand | Documents held speculatively | Latency at the point of use |
| Cap run length, checkpoint and restart | The tail of the curve entirely | Requires durable state to resume from |
The last one is the least popular and the most effective. If accuracy falls with length, a run that is split in two with a summary between the halves is not a workaround. It is the fix.
Note the tension with prompt caching. Compaction rewrites the front of the prompt, which invalidates the cache. Summarise too eagerly and you pay in recomputed tokens what you saved in accuracy. The compromise most teams reach is to compact on a step boundary rather than on a token threshold, so the rewrite happens rarely and predictably.
How would you know if this is happening to you?
Measure accuracy against context length, not accuracy overall.
Take a task your fleet performs often, run it at four context sizes spanning what you actually reach in production, and grade the outcome the way production grades it. If the curve is flat, you have headroom. If it bends, you have found your operating limit, and it is almost certainly lower than the number on your provider’s specification page.
The question for your next design review
Not “is our context window big enough”. Ask: at what context length does this task start failing, and how often do we exceed it? If nobody has plotted that curve, the window size in your configuration is a guess dressed as a limit.
Frequently asked questions
What is context rot?
The observed decline in model accuracy as input length increases, independent of whether the relevant information is present. Published evaluations report it across all major models tested. It is distinct from truncation: the information is in the window and the model still fails to use it.
Do larger context windows fix long-context degradation?
No. Reported findings are that million-token windows extend the point at which degradation appears rather than eliminating it. A larger window buys headroom and changes nothing about the shape of the curve.
Why do needle-in-a-haystack tests understate the problem?
Because they usually plant a target that closely matches the query wording, which is the easiest possible retrieval. Research reports that degradation accelerates as semantic similarity between query and target falls, so real tasks with paraphrased or implied targets fail earlier than the benchmark suggests.
How do you stop an agent's context from degrading over a long run?
Give it an eviction policy. Summarise completed steps rather than carrying their full transcript, keep tool outputs out of context unless a later step needs them, re-retrieve on demand instead of holding everything, and measure accuracy at the context lengths you actually reach rather than at the ones you tested.
Method and sources
- Classifier Context Rot: Monitor Performance Degrades with Context Length, arXiv:2605.12366. Source for degradation observed when monitoring long agent transcripts.
- Diagnosing and Mitigating Context Rot in Long-horizon Search, arXiv:2606.29718. Source for the observation that agentic context is multi-turn, multi-source and progressively accumulated, unlike single-turn benchmarks.
- Chroma research on context rot, evaluating performance degradation with increasing input tokens across Claude Sonnet 4, GPT-4.1, Qwen3-32B and Gemini 2.5 Flash, including the finding that degradation accelerates as semantic similarity between query and target decreases.
- Anthropic, Effective context engineering for AI agents, consulted for context management patterns.
- No figures in this article are our own measurements. Model names and the direction of the reported effect come from the published evaluations cited above.