THE ECONOMICS OF AGENTS

WHY INFERENCE IS USUALLY THE LARGEST COST AND LATENCY DRIVER Inference is the technical word for running a trained model to produce an answer. Training is when the model learns; inference is when the model is used. Every time your harness calls the model, that is one inference call. Latency means delay, the time between asking for something and receiving it. In most agents, inference is the largest single source of both cost and delay. There are two reasons. First, running a large model requires expensive specialized computer chips, and the provider passes that expense to you through token prices. Second, the model writes its answer one token at a time, so a long answer simply takes a long time to produce. Compare that to a tool call such as looking up a record in a database. A database lookup takes a few milliseconds and costs a tiny fraction of a cent. Even if your agent makes ten database calls in a run, their total cost is usually dwarfed by one or two inference calls. This has a practical consequence. When you want to save money or time, look first at the number and the size of the model calls, not at the tools. Making a database query twenty percent faster rarely matters. Removing one unnecessary model call, or shrinking the text sent to it, often matters a great deal.