LATENCY OPTIMIZATION

EXAMPLES Example 1: Measuring latency A support agent takes 12 seconds to answer. The engineer opens the trace and builds a breakdown. Model thinking before the tool call took 3 seconds, which is 25 percent. Waiting for a slow external shipping-status service took 7 seconds, which is about 58 percent. Model writing the final answer took 2 seconds, which is about 17 percent. The tempting first move would be to switch to a faster model. But even a model that was twice as fast would save only about 2.5 seconds. The real target is the external service. The engineer adds a cache for shipping status that lasts five minutes, and sets a 3-second timeout with a fallback message. The typical response time falls from 12 seconds to about 5 seconds. Example 2: Parallelism An inventory agent must check stock in three warehouses. Each check takes about 2 seconds. Run one after another, the checks take about 6 seconds. The checks do not depend on each other, so the harness launches all three at once and waits for all to finish. That portion now takes about 2 seconds, a cut of roughly two thirds. The engineer also adds a limit of ten simultaneous requests per agent, so a busy day with many users does not flood the warehouse system. Example 3: Streaming A report-writing agent takes 10 seconds to produce a long answer. Without streaming, the user sees a blank screen for 10 seconds and might think the page froze. With streaming, the first words appear after about a second, and the rest flow in as written. The total time is still 10 seconds, but the experience is completely different, because the user begins reading immediately. The engineer chooses to stream the narrative text, but to hold back a final "recommended action" box until it has passed a validation check. Example 4: Failing fast An agent calls a currency-conversion tool that usually answers in 400 milliseconds. One afternoon, the service becomes unresponsive. Without a timeout, the agent waits 60 seconds, the user gives up, and the interaction silently hangs. With a 3-second timeout, the call fails after 3 seconds. The harness then tries a backup rate source, which answers in one second. The user gets an answer in about five seconds, and the response includes a note saying a backup source was used.