LOGGING, METRICS, AND DASHBOARDS

CHOOSING WHAT TO LOG It is tempting to log everything, but a flood of messages hides the useful ones. Log events that matter for understanding behavior and for answering questions later. These include the start and end of each run, every model call and tool call, every decision to retry or stop, every safety block, and every error. Give each message a level, such as debug for fine detail, info for normal events, warning for odd but harmless events, and error for failures. Levels let you turn detail up when investigating and down when everything is quiet. Write messages in a consistent structured form so that tools can search them, and include the run identifier in every message. RATES, AVERAGES, AND PERCENTILES Numbers need careful reading. An average can hide a problem, because a few extremely slow runs can sit beside many fast ones. If nine runs take two seconds and one takes ninety seconds, the average is about eleven seconds, which describes none of the runs. A percentile shows the picture better. The ninety-fifth percentile is the time that ninety-five percent of runs beat, so it reveals the slow tail that users actually feel. Track rates as well as counts, such as the share of runs that fail, because a count of fifty failures means little until you know whether there were a hundred runs or a million.