INCIDENT RESPONSE FOR AGENTS

EXAMPLES Example 1: Detecting and triaging An automatic alert fires at 2:10 in the morning because the agent's error rate crossed 15 percent, above the 3 percent baseline. The on-call engineer opens the observability tools from Part Seven and filters traces by error. She finds that nearly all failures involve one tool that is returning timeouts. She checks how many users are affected, about 400 in the last twenty minutes, and notes that the agent is giving a fallback message rather than harmful output. She classifies the incident as medium severity, because service is degraded for many users but no data or money is at risk. She opens an incident record, writes the first timeline entries, and begins containment, all within about ten minutes. Example 2: Containing runaway behavior An agent enters an unexpected loop, calling the model repeatedly while trying to fix a formatting problem. Each pass uses thousands of tokens. The budget tracker built in Chapter 25 notices the run has exceeded its per-run spending limit and pauses it automatically, within seconds. The total cost is a few dollars instead of the hundreds it could have been over an unwatched weekend. An engineer later examines the paused run, identifies the loop, and fixes the cause. This shows why limits enforced by the harness are a form of containment that works even when nobody is awake. Example 3: A blameless postmortem An agent sent customers wrong refund amounts for three hours after a prompt change. The postmortem summary explains this in plain terms. The timeline shows when the change was released, when the first complaint arrived, and when the rollback happened. The root cause section does not say that an engineer made a mistake. It says the prompt change was released without being run against the refund test cases, because the release pipeline had no required evaluation step, and because no alert existed for refund amounts outside the normal range. Contributing factors list the missing automated gate, the missing alert, and an ambiguous ownership of the prompt file. Action items add a mandatory evaluation gate (owner: placeholder, due: placeholder), add an alert on unusual refund amounts, and assign an owner for prompt changes. The engineer who deployed the change helps write the review and is thanked for being open about what happened. Example 4: A feedback loop After the refund incident, the team takes the actual inputs that triggered the failure and turns them into three new test cases in the evaluation suite. These tests check that the agent calculates refund amounts correctly in the same situations that failed. From now on, every release runs them automatically. Six months later, a developer changes the prompt again, and the pipeline fails on those tests before the change reaches users. The regression is caught, because the old failure had become a permanent test.