WHY OBSERVABILITY MATTERS

WHAT TO CAPTURE ON EVERY RUN Even a small harness should capture a few basics on every run. Capture a unique identifier for the run so that every record can be tied together. Capture the start and end times, so that you can measure how long the run took. Capture each model call with its input size, output size, and the name of the model. Capture each tool call with its name, its arguments, its result or error, and how long it took. Capture the final outcome, whether success, failure, or a handoff to a human, and the reason. Capture the cost, counted in tokens or in money. With only these items you can answer most of the questions engineers ask on a bad day. PRIVACY AND SAFETY OF OBSERVABILITY DATA Observability data can contain sensitive material, because prompts and tool results often include personal details or secret keys. Treat the data as sensitive by default. Redact, meaning remove or mask, secrets and personal details before they are stored. Limit who can read the data, and delete it after a reasonable period. Never write passwords or access keys into logs, even by accident. A common safeguard is to log the names and sizes of things and to store full contents only for a small, protected sample.