THE EVALUATION HARNESS

WHY YOU NEED ALL FOUR LEVELS Each level catches problems that the others miss. An agent can pass every span check, because each individual step looked reasonable, and still fail the outcome, because the steps added up to the wrong result. An agent can also reach the correct outcome by a dangerous or wasteful path, which only span-level and trace-level checks would reveal. Good evaluation therefore measures several levels at once, much as a doctor looks at your pulse, your blood pressure, and your overall energy rather than relying on only one number. WHERE THE TEST INPUTS COME FROM An evaluation harness needs inputs, and there are two main sources. INPUTS FROM REAL PRODUCTION TRACES Production means the live system that real users are actually using, as opposed to a practice version. Because your observability layer records traces from production, you already own a rich collection of real requests. You can pick traces from that collection and turn them into test cases. The great strength of this source is realism. Real users ask surprising things, make typos, mix languages, and phrase requests in ways no designer would have imagined. The weakness is that real traces are messy, they may contain private information that must be removed before you reuse them, and you may not know what the correct answer should have been.