DEPLOYMENT ARCHITECTURES

SCALING TO MANY CONCURRENT USERS Concurrency means many things happening at once. A harness built for one user can fail when ten thousand arrive at once. Scaling means growing capacity to meet demand. Vertical scaling means using a bigger machine, while horizontal scaling means using many machines side by side. Horizontal scaling is usually preferred for agents because it can grow without limit and survives the loss of any single machine. Key techniques include the following. Stateless workers: if each worker keeps no private memory between requests and stores session data in a shared database, any worker can handle any request. Queues: a queue is a waiting line for work. When requests arrive faster than they can be handled, they wait in line rather than being lost, and workers pick them up as they become free. Connection pooling: opening a connection to a database or service is slow, so a pool keeps a set of ready connections to be shared. Isolation: one slow or stuck session must not block the others, so sessions should run independently with their own time limits. Rate-limit management: model providers cap requests and tokens per minute, so the harness must share that limit fairly among users and slow down gracefully when near the cap. Autoscaling: adding and removing workers automatically based on load. Agents have special scaling issues. A single agent run can last minutes, tying up resources much longer than a typical web request. Runs may need to be paused and resumed, which requires saving their state. Cost scales with usage, so you also need per-user budgets to stop one person from consuming everything.