CONTEXT ROT AND CONTEXT BLOAT

EXAMPLES Example 1: Bloat from a verbose log. An agent is asked to run a test suite and fix whatever fails. The test tool prints 4,000 lines of output, nearly all of them lines saying "test passed." Buried in the middle is a single line: "FAILED: test checkout total, expected 49.99 got 50.00." The careless harness appends all 4,000 lines to the context, using perhaps 30,000 tokens. Now the window is crowded with passing lines that no longer matter, and there is less room for the actual code the agent needs to read next. A smarter harness filters the tool output before appending it. It keeps only the failing lines plus a one-line summary such as "4,000 lines; 1 failure." It saves the full log to a file in case it is needed. The context stays lean, and the one important line is easy to find. Example 2: Rot in a near-maximum window. Suppose a model has a window of 200,000 tokens, and you fill it nearly to the limit with a long report. Near the middle of the report, a sentence says that the contract renewal deadline is March 3. You ask, "What is the contract renewal deadline?" With a short version of the report, containing only a few pages around that sentence, the model answers March 3 every time. With the full near-maximum version, the model sometimes answers a different date that appears elsewhere in the document, or says it cannot find one. Nothing about the fact changed. Only the surrounding length changed. That is context rot: accuracy degrading as length grows, even within the technical limit. A harness that notices long contexts could respond by retrieving just the relevant section before asking the question. Example 3: A needle-in-a-haystack test, step by step. Let us design one together, using the secret code idea. Step one: prepare three haystacks made of ordinary text, one of 5,000 tokens, one of 50,000 tokens, and one of 100,000 tokens. Step two: for each haystack, make three copies, and insert the sentence "The secret code is 4471" near the start in the first copy, in the exact middle in the second copy, and near the end in the third copy. That gives nine test documents. Step three: for each document, ask the model, "What is the secret code?" Step four: record the answer as correct or incorrect in a grid. Step five: study the grid. Suppose all nine pass at 5,000 tokens, all pass at 50,000 tokens, but at 100,000 tokens the middle-position copy fails. You have learned that your model is trustworthy up to about 50,000 tokens for this kind of fact, and that you should keep important material away from the middle or keep the window under that size. Notice that you answered a business question with a small, cheap experiment. Example 4: A safety instruction that fades. An agent is told in its opening instructions never to send emails to people outside the company without approval. In short sessions it obeys perfectly. But in a session with several hundred messages, the agent drafts and sends an email to an outside address with no approval request. The rule has not been deleted. It is still sitting in the opening instructions, but it is now very far from the most recent text, and the model's attention to it has weakened. This is a serious symptom, because safety rules are exactly the things we cannot afford to lose. The fix is not to hope. The fix is to design the context builder to re-state critical rules near the end of the context, using short system reminders, which you will learn in Chapter 20, and to enforce the rule in code through hooks, which you will learn in Chapter 24, so that the safety does not depend on the model's memory alone.