LOGGING, METRICS, AND DASHBOARDS

AVOIDING ALERT FATIGUE An alert is a message that tells a person something needs attention. If alerts fire too often or for trivial reasons, people stop reading them, which is called alert fatigue. A good alert is urgent, specific, and actionable, and it links to the runbook that explains what to do. Alert on symptoms that users would feel, such as a spike in failures, rather than on every small internal wobble. Review alerts after each incident, and remove or tune the ones that never lead to action. A STARTER DASHBOARD A first dashboard can be very small. Show the number of runs, the success rate, the average and slowest times, the average cost, and the number of runs handed to humans, each over the last day and the last week. Add a list of the most common errors, so that the biggest problem is always visible. Put the dashboard where the team will see it every morning. Charts are only useful when somebody looks at them and acts, so build the habit as well as the screen.