HOW LLMs WORK FOR THE NON-SPECIALIST

TOKENS A token is a small chunk of text that a model reads and writes. Models do not work directly with letters or whole words. Before text reaches the model, a program called a tokenizer chops it into tokens, and each token is replaced by a number that the model can work with. A token can be a whole common word, a piece of a longer word, a single punctuation mark, or even a space combined with a word. As a rough rule of thumb for English, one token is about three quarters of a word, so one hundred tokens come to about seventy-five words. This is only an estimate, and it differs between models and between languages. Computer code, rare words, and some non-English languages often take more tokens per word. Why do tokens matter to a harness engineer? They matter because nearly everything about running a model is measured in tokens. The cost of using a hosted model is usually charged per token. The speed of a response depends on the number of tokens produced. And the limit on how much the model can consider at once is also a token limit. A harness engineer who ignores tokens will be surprised by bills, delays, and cut-off answers.