LATENCY OPTIMIZATION

PARALLELISM AND STREAMING RESULTS Parallelism means doing several things at the same time instead of one after another. If three tasks each take 2 seconds and do not depend on each other, doing them in sequence takes 6 seconds, but doing them in parallel takes about 2 seconds. Imagine three cooks each preparing a different dish at once, rather than one cook making all three dishes in turn. The important condition is independence. If task B needs the result of task A, B must wait. Tasks that do not depend on each other, such as looking up three separate records, can run side by side. Many models can ask for several tool calls in one turn, and a harness can run them concurrently, which means overlapping them in time. Be careful with parallelism. Running a hundred requests at once might exceed a rate limit or overload a service. Use a cap on how many run at once. Also take care when parallel actions change shared data, because they could conflict. Streaming means sending partial results to the user as they are produced, rather than waiting for the complete result. When you watch a chat answer appear word by word, you are seeing streaming. Streaming does not make the model finish sooner. It improves perceived responsiveness, because the user sees progress immediately and can start reading. Streaming has design questions. Some output should not be shown until it is checked, for example an answer that must pass a safety review first. Streaming a half-formed answer that is later retracted can confuse users. So decide, step by step, what to show live and what to hold until it is ready. Progress messages, such as "searching the job database," are a good middle path, because they reassure the user without exposing unverified conclusions.