AGENT OPERATIONS (AGENTOPS)

CONTINUOUS MONITORING, DRIFT DETECTION, AND ALERTING Monitoring means continuously collecting and watching measurements about the running system. Useful measurements for agents include success rate, error rate, cost per task, tokens per task, number of tool calls per task, time per task, rate of escalation to humans, user ratings, and the rate of policy blocks. Drift means a slow change in behavior or performance over time without any obvious single cause. Agents drift for many reasons. The model provider may update a model version. The kinds of requests users make may change. Data the agent depends on may change. Small prompt edits may pile up. Drift is dangerous because no single run looks broken, yet things get steadily worse. To detect drift, establish baselines, meaning normal ranges for each measurement, and compare current numbers against them. An alert is an automatic notification when a measure leaves its normal range. Good alerts are actionable, meaning they tell a person what is wrong and where to look. Beware alert fatigue: if alarms fire constantly for trivial reasons, people start ignoring them, including the important ones. Continuous evaluation means regularly running test cases against the live system or a sample of its real outputs, to measure quality and not only speed and cost.