PLATFORMS AND MANAGED SERVICES

EVALUATION AND OBSERVABILITY PLATFORMS Part Seven taught observability, which means being able to see inside the agent through traces, logs, and metrics. Part Eight taught evaluation, which means measuring how well the agent does its job. An evaluation and observability platform is a third-party service that collects your agent's traces and gives you dashboards, searching, alerts, and tools for running and tracking evaluations. Recall from Chapter 28 that a trace is the step-by-step record of one agent run. A dashboard is a screen of charts and numbers that summarizes what is happening. You plug your harness into such a platform by sending it the traces, usually by adding a few lines of code or by using a standard telemetry format. Telemetry is information automatically sent from a running system to a place where it can be studied. An open standard called OpenTelemetry, widely used for collecting traces and metrics, is supported by many such services, which makes it easier to switch between them. Benefits include the following. Speed: you get useful dashboards in hours rather than weeks. Team features: shared views, comments, and permissions for people who are not programmers. Experiment tracking: comparing evaluation scores between versions of your agent. Alerting: notifications when errors or costs spike, which feeds the Agent Operations (AgentOps) routines of Chapter 59. Risks include the following. Data sensitivity: traces often contain customer text, personal information, and secrets that leaked into prompts, so you must understand what the service stores and for how long. Cost at scale: trace storage grows with your traffic. Dependence: if your evaluation history lives only in one vendor's system, leaving is painful. Redaction: remove or mask sensitive fields before sending traces out whenever you can, using the privacy practices from Part Nine.