The problem
Your support agent is sharp for the first ten messages of a conversation. By message forty it is contradicting itself, forgetting what the customer already told it, and citing a policy that was superseded twenty messages ago.
I see a lot of companies discover this the hard way, usually in a long running agent that worked fine in testing because nobody tested a long conversation.
The instinct is to blame the model or reach for a bigger window. After 4 years building these systems, I think that instinct is exactly backwards.
Context rot is a measurable thing
This is not folklore. Chroma's research on it is the clearest: model performance degrades as input token count increases, on tasks the same model handles perfectly with a smaller, well curated context. It held across Claude Sonnet 4, GPT-4.1, Qwen3-32B and Gemini 2.5 Flash. Not a quirk of one vendor.
Google DeepMind found a related effect on position. Information sitting in the middle of a long window is recalled 20 to 30% less accurately than the same information at the beginning or the end.
Put those together and the conclusion is uncomfortable for how most people build:
A 200,000 token window is not 200,000 tokens of reliable working memory. It is a budget you can overspend, and overspending degrades everything in it, including the parts you care about.
Why a bigger window makes it worse
When the window grows, teams fill it. That is just what happens.
The logic feels sound. We have room, so let us include the full conversation history, the whole knowledge base article, the complete CRM record, the last six tickets. More information means better decisions.
Except the model is not searching a database. It is attending over everything you gave it, and attention is finite. Every irrelevant token you add competes with the relevant ones.
You did not give it more to work with. You lowered the signal to noise ratio and pushed the important line into the middle, which is the worst place for it.
How I budget a window
I treat the context window like a payload with a weight limit, not a bucket. Everything in it has to justify its seat.
Rank by decision relevance, not availability
The question is never "do we have this data." It is "does this change what the agent should do right now."
A customer's full order history is available. For a password reset question it is noise. For a refund decision it is the whole ballgame. Same data, different call, different answer.
Put the decisive content at the edges
Given the position effect, I place the task instruction and the single most decision relevant piece of information at the start and the end. Reference material that is nice to have goes in the middle, because that is where degradation is cheapest.
Compact instead of accumulate
For anything long running, raw history is the enemy. Past a threshold I summarize the conversation into a structured state object: what the user wants, what has been established, what has been ruled out, what is still open.
That object is a few hundred tokens and it carries more usable signal than the 8,000 tokens of transcript it replaced. Then the transcript gets dropped.
This is the single highest leverage change I make to long running agents, and it usually fixes the decay outright.
Retrieve narrowly
Do not fetch the document. Fetch the passage. If your retrieval step returns 4,000 tokens to answer a question that needed 200, your retrieval step is the problem, not the model.
Budget per call, and enforce it
I set an actual token ceiling for each agent call and instrument against it. When a call goes over, something gets cut, and the decision of what gets cut is made at design time by me rather than at runtime by a truncation function.
The test that exposes this
Take a task your agent handles correctly with minimal context. Then run the identical task with 30,000 tokens of plausible but irrelevant material in front of it.
If accuracy drops, and it usually does, you have measured your own context rot. Now you know the ceiling you are actually operating under, and you can design for it.
Most teams have never run this test, which is why they are surprised when a long conversation goes sideways.
What this changes in practice
The shift is from "what can we give the model" to "what is the smallest set of high signal tokens that makes this decision correct."
That is a harder engineering question. It means knowing the decision well enough to know what is actually load bearing. It is also the difference between an agent that holds up over a forty turn conversation and one that quietly falls apart after ten.
More context is not more intelligence. Past a point it is measurably less.
I design and maintain the context layer for the AI systems I build, because this is where most production agents actually fail. If you have an agent that starts strong and degrades, this is almost always what is happening.