HOW LLMs WORK FOR THE NON-SPECIALIST

THE CONTEXT WINDOW The context window is the limited amount of text, measured in tokens, that a model can consider at one time. Think of it as the model's desk. Everything that is on the desk, meaning your question, the instructions, the earlier conversation, and any documents you attached, can be seen and used. Everything that is not on the desk does not exist for the model, no matter how important it may be. The window includes both what you send in and what the model writes back, so a long answer uses up some of the same space. Context windows have grown over the years. Some modern models support windows of hundreds of thousands of tokens, and a few support a million or more. A window of two hundred thousand tokens corresponds very roughly to a five hundred page book, though the exact amount depends on the text. A bigger window is helpful, but it is not a cure for everything. Models can pay less careful attention to material in the middle of very long inputs. Filling the window also costs more money and takes more time. And an overloaded window can bury the important facts under unimportant ones. Deciding what goes on the desk is such a big topic that it has its own name and its own Part of this book: context engineering, introduced in Chapter 4 and treated in depth in Part Five.