CONTEXT ROT AND CONTEXT BLOAT

CONTEXT ROT Now we come to a more surprising problem. Context rot means that a model's accuracy drops as the window grows very large. The word rot suggests slow decay, like fruit left on a counter. The surprising part is that the decay can happen even when the text still technically fits inside the model's stated limit. You might expect that a model with a very large window would treat all of that text equally well. In practice, researchers and engineers have repeatedly observed that this is not the case. Models tend to be most reliable when relevant information is present in a modest amount of text, and they can become less reliable at finding, weighing, and using the right detail as the amount of surrounding text grows. A well-known research paper from 2023 titled "Lost in the Middle" reported that models often use information at the very beginning and the very end of a long input better than information in the middle. More recent industry write-ups, including a 2025 technical report from the vector-database company Chroma that popularized the phrase "context rot," tested many models and found that performance generally varies and often declines as input length grows, even on simple tasks. I mention these sources so you know the idea comes from observation and measurement, but you should not memorize any particular percentages. Different models behave differently, new models keep improving, and the safe lesson is this: never assume that a larger window means equally good performance across the whole window. Always measure your own harness with your own model. Why might rot happen? A plain-language explanation is that the model must decide which parts of a long text to pay attention to, and the more text there is, the more places its attention can be pulled toward things that merely look similar to what it needs. Imagine looking for your friend's face in a small room versus a stadium. In the small room, you find them at once. In the stadium, you might mistake a stranger who looks similar. The detail is still there, but finding it reliably is harder. Researchers sometimes call these look-alike items distractors, meaning content that resembles the answer but is not the answer. More text usually means more distractors.