CONTEXT AS THE SCARCE RESOURCE

A SIMPLE CONTEXT BUDGET Let us walk through a made-up budget so that this feels concrete. Imagine a model with a context window of 100,000 tokens. That is the entire desk. An engineer might divide it like this. First, reserve about 5,000 tokens for the system instructions and tool descriptions, because these must always be present. Second, reserve about 10,000 tokens for the model's answer, so that the reply is never cut off. Third, allow about 30,000 tokens for recent conversation history. Fourth, allow about 40,000 tokens for retrieved documents and tool results. Fifth, keep about 15,000 tokens as a safety margin, because token counts are estimates and surprises happen. Add these up and you get 100,000 tokens. The exact numbers are not magic. The important point is that every category has an allowance, and the harness checks usage against the allowances before each model call. When a category goes over its allowance, the harness must act, for example by summarizing old history or trimming a long tool result. This habit of checking usage before each call is called a context-budget check, and building one is the lab exercise at the end of this chapter. Notice also that the answer space is reserved first. The model's reply must fit inside the same window as the input, so if your input fills the window completely, the model has no room to answer. Engineers who forget this see mysterious cut-off answers or outright errors.