LATENCY OPTIMIZATION

CHAPTER SUMMARY Measure first: break total time into inference, tools, network, queueing, and overhead, and attack the biggest slice. Parallelism overlaps independent work, with caps to protect services. Streaming improves perceived speed but needs decisions about what is safe to show early. Caching plans and intermediate results saves time but requires expiry rules to avoid stale data. Timeouts let the agent fail fast and move to a fallback. KEY TERMS REVIEW Latency: delay before a result. Time to first token: wait before output begins. Percentile: a value below which a given percent of measurements fall. Rate limit: a cap on requests per period. Parallelism: doing independent tasks at the same time. Streaming: sending output as it is produced. Cache: stored results for reuse. Stale data: saved information that is no longer true. Timeout: a limit on how long to wait.