FROM MODEL TO APPLICATION

INFERENCE Inference means actually running a trained model to get an answer. It contrasts with training, which is the earlier, very expensive process of creating the model. Every time your harness sends a request and receives a completion, one act of inference has occurred. You can think of training as writing and printing a cookbook, and inference as cooking a single meal from it. Inference is where your costs and delays show up. The time it takes depends on the model's size, the length of the input, the length of the output, how busy the provider is, and whether the model does extra internal reasoning before answering. Some modern models are able to spend additional effort thinking before they answer, which can improve results on hard problems but also adds time and tokens. Many APIs also support streaming, in which the answer arrives piece by piece as it is generated rather than all at once. Streaming makes a chat feel faster, because the reader sees words appear immediately, even though the total time is the same. Harness engineers choose between streaming and waiting depending on whether a human is watching.