CONTEXT ROT AND CONTEXT BLOAT

NEEDLE-IN-A-HAYSTACK TESTING How do we find out whether a particular model degrades on long inputs? One widely used method is called a needle-in-a-haystack test. The name comes from the saying about finding a needle in a haystack. Here is how it works. First, you take a very long piece of ordinary text, the haystack. Second, you insert one specific, easy-to-check fact somewhere in it, the needle. An example needle is the sentence "The secret code is 4471." Third, at the end, you ask the model a direct question, such as "What is the secret code?" Fourth, you check whether the answer is correct. Then you repeat the experiment many times, changing two things. You change the length of the haystack, from short to very long. And you change the position of the needle, from the beginning to the middle to the end. You record whether the model found the needle each time. The results are often drawn as a grid, with length on one side and position on the other, and each cell marked as success or failure. Such a grid reveals where the model's retrieval reliability weakens. Perhaps it is perfect up to a certain length and then fails whenever the needle sits in the middle. That knowledge is valuable, because it tells you how large and how organized to make your contexts. There are important cautions. The classic version of this test uses a single, easy, obvious fact, so passing it does not prove the model is good at real work with many related facts and subtle questions. Think of it as a minimum safety check, like checking that a car's brakes work in an empty parking lot before trusting it on a busy highway. A model that fails even this simple test is clearly risky, but a model that passes still needs testing on tasks that resemble your real work. For this reason, engineers build domain-specific versions, using needles and haystacks that look like their actual data, such as a particular skill hidden in a long resume bundle. You will use exactly this idea in the lab.