CONTEXT AS THE SCARCE RESOURCE

THE OPERATING-SYSTEM ANALOGY There is an even richer comparison that engineers like to use. Think about the layers inside a normal computer. At the center is the CPU, the chip that actually performs calculations and makes decisions. Next to it is the RAM, which holds what the CPU is working on right now. Beyond that are the disk drives, which hold much larger amounts of information for the long term but are slower to reach. And managing all of this is the operating system (OS), the master program that decides which data sits in RAM, which stays on disk, and when to move things back and forth. Now map the agent onto this picture. The model is the CPU, because it does the thinking. The context window is the RAM, because it holds what the model is thinking about right now. External storage such as files and databases is the disk, because it holds far more but must be fetched. And the harness, especially the part we call the context builder, is the operating system, because it decides what gets loaded into the window and what stays out. A context builder is the piece of the harness whose job is to assemble the exact text the model will see on each call. We will study it in detail in Chapter 20. For now, notice the lesson of the analogy. The CPU does not decide what is in RAM. It simply works on whatever it is given. In the same way, the model does not usually decide what is in its context. The harness does. This means the quality of an agent depends heavily on how good the harness is at being a memory manager, not only on how clever the model is. A brilliant model starved of the right information, or buried under the wrong information, will perform badly.