LOGGING, METRICS, AND DASHBOARDS

A WORKED METRICS EXAMPLE Consider an agent that handled one thousand runs this week. Nine hundred and twenty succeeded, fifty failed on a tool error, and thirty were handed to people. The success rate is therefore ninety-two percent, the tool failure rate is five percent, and the handoff rate is three percent. Last week the tool failure rate was two percent. That change from two to five percent is the signal worth chasing, and a chart over time would have shown the climb beginning on Wednesday. Open the traces for a few of the failed runs on Wednesday and Thursday, find the common tool, and look at what changed that day. The metric told you where to look, and the trace told you why. Good teams use both together. LOGS, METRICS, AND PRIVACY All three kinds of data raise privacy questions. Metrics are the safest because they are only numbers. Logs and traces may hold the words of users, so apply the redaction and retention rules described in Chapter 27. Decide in advance how long each kind of record is kept, who can read it, and how it is deleted. Writing these decisions down is part of responsible operation. COST OF MONITORING ITSELF Monitoring is not free. Storage, processing, and the time of the people who read dashboards all cost money. Avoid tracking hundreds of numbers that nobody looks at. Choose a small set tied to the questions you actually ask, and add more only when a real decision needs them.