LATENCY OPTIMIZATION

MEASURING AND REDUCING RESPONSE TIME You cannot improve what you cannot see. Before changing anything, measure where the time goes. Recall from Chapter 28 that a trace records a run step by step, and good traces include timestamps showing when each step started and ended. Total response time is usually made of several parts. Model inference time: how long the model takes to read the prompt and write its output. This depends on how long the prompt is and, much more, on how long the output is, since the model writes one token at a time. Tool-call time: how long each tool takes to do its work, such as querying a database or calling an outside service. Network time: the time messages spend traveling between machines. Queueing time: the time a request waits in line before anything starts, for example because a service is busy or a rate limit is holding it back. A rate limit is a cap on how many requests you may make in a period of time. Harness overhead: the time your own code spends preparing and processing data. A helpful measurement is a latency breakdown, a table listing each component, its time, and its percentage of the total. The biggest percentage is your target. Another useful measure is time to first token, the wait before the first piece of output appears, which matters more to how fast the agent feels than the total time. Averages can hide trouble. Look also at the slowest cases, often described as percentile numbers. The 95th percentile means that 95 percent of runs were faster than this value and 5 percent were slower. A system with a good average but a terrible 95th percentile will frustrate one in twenty users. Common ways to reduce time include shrinking prompts, since less input is processed faster, asking for shorter outputs, routing simple tasks to smaller faster models as in Chapter 54, using prompt caching, and removing unnecessary steps.