CONTEXT AS THE SCARCE RESOURCE

WHY CONTEXT IS FINITE Why is the window limited at all? There are several reasons, and it helps a beginner to understand them in plain terms. First, there is a technical limit built into each model. When a model is created, it is designed and trained to handle sequences up to a certain length. That length is the maximum context window of that model. Different models offer different sizes, and the sizes have grown over the years, but there is always a ceiling. Second, there is a cost limit. Most model providers charge by the token, counting both what you send in and what the model writes out. A bigger context means more tokens sent on every call, and since an agent makes many calls in a loop, the cost multiplies quickly. Think of a taxi that charges by the distance and you must take many trips. Carrying a heavy suitcase of unnecessary papers on every trip makes each trip more expensive. Third, there is a speed limit. The more text the model must read before it can answer, the longer it takes. An agent stuffed with unneeded text answers more slowly. Fourth, and this is a subtle point we will explore fully in Chapter 19, there is a quality limit. Even when text technically fits within the window, models tend to become less reliable at using information when the window is very full. So a bigger desk does not mean you should pile papers all over it. Because of these four limits, context must be budgeted carefully. Budgeting means deciding in advance how much of a limited resource each purpose is allowed to use. A household that earns a fixed amount each month decides how much goes to rent, food, and savings. An agent harness does the same with tokens.