CONTEXT AS THE SCARCE RESOURCE

WHAT IS A CONTEXT WINDOW Let us begin from zero. An LLM, as you learned in Chapter 2, reads the text you give it and writes new text in response. When you talk to an LLM, you do not talk to it directly in the way you talk to a friend. Instead, a piece of software collects everything the model should know for this moment, packs it into one long block of text, hands it to the model, and receives the model's reply. That one long block of text is called the context. The context window is the maximum size of that block. You can picture the context window as a desk. Everything the model is currently working on must be laid out on top of that desk. The desk has a fixed size. If you try to put more papers on it than it can hold, something has to be moved off, or the papers simply will not fit. Here is another key idea. The model has no memory of its own between calls. Each time the harness calls the model, the model starts completely fresh, like a person with no recollection of yesterday who only knows what is written on the desk in front of them. So if the harness wants the model to remember that the user asked for a summary in French, the harness must place that fact on the desk again every single time. The size of the desk is measured in tokens. A token is a small chunk of text, roughly a short word or a piece of a longer word. As a rough rule of thumb in English, one token is about three quarters of a word, so one thousand tokens is roughly seven hundred and fifty words. This is only an approximation, and different models count tokens slightly differently, so a careful engineer always measures with the real tokenizer (the tool that splits text into tokens) of the model being used. What lives inside the context window? Several different things share the space. First, there are the system instructions, which are the standing rules and role description the harness gives the model, such as "You are a careful resume reviewer." Second, there are the tool descriptions, which tell the model what tools it may call and how to call them. Third, there is the conversation history, meaning the messages exchanged so far between the user, the model, and the tools. Fourth, there are the tool results, meaning the text that came back when the agent ran a tool, such as the contents of a file or the output of a web search. Fifth, there are any retrieved documents, meaning extra reference material the harness looked up for this particular question. Sixth, there must be room left over for the model's own answer, because the answer is also measured in tokens and it too must fit. That sixth item is the one beginners forget most often, and we will return to it.