COST OPTIMIZATION TECHNIQUES

EXAMPLES Example 1: Prompt compression A team has a 500-word system prompt for a support agent. It contains the sentence "Always be polite" in five different forms, three long paragraphs of company history that never affect any answer, and eight nearly identical examples. An engineer rewrites it into a 150-word version. The new version has eight short numbered rules, one example, and the one-line company name. Using the earlier rough rule that a word is about 1.3 tokens, the old prompt is about 650 tokens and the new one is about 200 tokens, saving 450 tokens per call. At 100,000 calls a month, that is 45 million tokens saved. At the made-up price of 3 dollars per million input tokens, that saves about 135 dollars a month from one edit. Before shipping, the engineer runs the 200-case evaluation suite. The score is 94 percent before and 94 percent after, so the change is kept. Example 2: Model routing An email assistant has two kinds of work. The first is "classify this email as spam or not spam," which is simple and happens thousands of times a day. The second is "research this customer's long-running problem across the last six months of tickets and propose a plan," which is complex and happens a few dozen times a day. The engineer routes classification to a small, cheap, fast model, and routing research to a larger, more capable one. Suppose the small model costs one tenth as much per token. If classification is 90 percent of calls, routing it to the small model cuts the cost of those calls by 90 percent, while the research quality is untouched. The engineer also adds an escalation rule: if the small model's answer is marked uncertain, the email goes to the larger model for a second look. Example 3: Prompt caching An agent for a software team has a system prompt plus tool definitions totaling 12,000 tokens, which never change. A typical session has 30 calls, and each call adds a short new message of about 500 tokens. Without caching, the unchanged 12,000 tokens are billed at the full rate 30 times, which is 360,000 tokens. With caching, the first call pays full price, perhaps with a small surcharge for storing the cache. The other 29 calls pay a much lower cache-read rate for the 12,000 tokens. If the cache-read rate is one tenth of the normal rate, the effective charge is 12,000 plus 29 times 1,200, which is 46,800 token-equivalents instead of 360,000, a saving of nearly 87 percent on that block. Calls are also faster because the provider skips reprocessing. The engineer makes sure the changing date and user name sit at the end of the prompt, not the start, so that the cached beginning stays identical. Example 4: Cutting wasted tokens A code-reading agent loads a 2,000-line file, about 25,000 tokens, into the conversation. Because of a design flaw, on each of its 10 steps it reads the file again and adds another copy to the conversation. By step 10, the conversation holds ten copies, and the agent has paid for 250,000 tokens of file text, plus the growth in every later call. The engineer changes the harness so the file is read once, stored, and referenced by name. That eliminates nine of the ten copies, a reduction of 90 percent of that redundant spend. The agent also becomes faster and, because its conversation is less cluttered, often more accurate.