CONTEXT AS THE SCARCE RESOURCE

EXAMPLES Let us walk through four scenarios in detail. Example 1: The computer with too many programs open. Picture a laptop with a small amount of RAM. You open a browser with twenty tabs, then a video editor, then a game. The laptop starts to crawl, and the operating system begins closing or pausing the least recently used programs to free memory. An agent faces the same pressure. Suppose a customer-support agent has been chatting for two hours and the conversation history now fills most of the window. A new message arrives from the customer. The harness has two choices. It can refuse to continue, which is a bad experience, or it can make room. To make room, it summarizes the oldest messages, perhaps the first hour of friendly small talk and a resolved shipping question, into a short note. It keeps the recent messages in full. Now the new message fits, and the agent continues smoothly. This is exactly like the operating system closing programs you have not touched in a while. Example 2: The research agent that read too much. Imagine a research agent asked to compare ten electric cars. The agent fetches 50 full web pages, and each page, after removing formatting, is about 3,000 tokens. That is 150,000 tokens just from reading. If the model's window is 128,000 tokens, the pages alone already overflow the desk. Even if the window were large enough to hold all of it, there would be almost no space left for the model to write its final comparison. The lesson is that context must be budgeted like a scarce resource and not treated as unlimited. A better design would have the agent read one page at a time, write down the key facts in a short note, discard the full page, and move on. After fifty pages, the agent would hold fifty short notes, maybe 200 tokens each, for a total of 10,000 tokens, which fits easily with plenty of room to write the answer. Example 3: The operating-system analogy in action. Consider a coding agent fixing a bug. The model is the CPU, doing the thinking. The context window is the RAM, currently holding the error message, the two files most relevant to the bug, and the plan. The project on disk may contain thousands of files, but they are not in RAM. The context builder acts like the operating system's memory manager. When the model says, "I need to see the file that defines the login function," the harness fetches that file from disk and loads it into the window. When the window starts to fill, the builder evicts, meaning removes, files that are no longer needed. The model never has to know about this housekeeping. It simply sees a well-organized desk each time. Example 4: The 300-page manual. Suppose an agent is asked a single simple question, such as "How do I reset the device to factory settings?" A careless design pastes the entire 300-page manual into the context. That might be 150,000 tokens, nearly all of which is irrelevant to resetting a device. The cost is high, the answer is slower, and the model is more likely to get distracted. A better design uses a retrieval step. Retrieval means searching a body of documents and pulling out only the pieces most related to the question. You will study retrieval in detail in Part Eleven. With retrieval, the harness finds the two paragraphs about factory reset, puts only those in the window, and the answer is faster, cheaper, and usually more accurate.