HOW LLMs WORK FOR THE NON-SPECIALIST

TEMPERATURE AND SAMPLING When the model scores every possible next token, it does not always have to pick the top one. The settings that control how it chooses are called sampling settings. The best known is temperature, a number that controls how random or adventurous the model's choices are. Picture a bag of lottery tickets in which likelier tokens have more tickets. With a temperature near zero, the model nearly always picks the token with the most tickets, so its answers become very consistent and predictable. With a higher temperature, the unlikely tokens get a better chance, so the answers become more varied and creative, but also riskier. Setting temperature to zero makes the model give nearly the same, most likely answer each time. That is helpful for factual lookups, data extraction, and anything where consistency matters. Note that some hosted systems do not produce perfectly identical output even at zero, and some newer models fix or restrict the setting, so check the documentation for the model you use. Setting the temperature to something like 0.9 gives more varied output, which suits brainstorming names, slogans, or story ideas. A related setting is top-p, sometimes called nucleus sampling, which limits the choice to the smallest group of likely tokens whose combined chance reaches a chosen amount. You do not need the details yet. What you need is the idea that these knobs trade consistency against variety, and that a harness should set them on purpose for each task rather than leaving them at defaults.