THE EVALUATION HARNESS

CHAPTER SUMMARY An evaluation harness is the system that feeds test cases to an agent, records what happens, scores the results, and reports them. Agents can be measured at four levels of granularity: spans for single steps, traces for whole runs, sessions for whole conversations, and outcomes for final results in the world. Test inputs come from real production traces, which are realistic but messy, and from curated datasets, which are controlled and stable. Offline evaluation tests a new version before release on fixed tasks, and online evaluation scores a sample of live traffic after release; strong teams use both and agree on a rollback rule in advance. KEY TERMS REVIEW Evaluation harness: the system that measures agent quality. Eval: short for evaluation, a structured test of quality. Span: one small unit of work, such as a single tool call. Trace: the full record of one run, made of spans. Session: a whole conversation or working period containing many runs. Outcome: the final real-world result of the agent's work. Production: the live system used by real users. Curated dataset: a deliberately chosen and maintained set of test cases. Offline evaluation: testing before release on fixed tasks. Online evaluation: scoring live traffic after release. Rollback: returning to the previous working version.