THE EVALUATION HARNESS

THE FOUR LEVELS OF GRANULARITY The word granularity means how fine or coarse your zoom level is. Think of looking at a city on a map. You can zoom all the way in to see one single house, zoom out a little to see a street, zoom out further to see a whole neighborhood, or zoom all the way out to judge whether the city is a good place to live. Each zoom level answers a different question. In Chapters 27 through 30, you learned that the observability layer records what an agent does. Evaluation uses those same records and measures them at four levels: spans, traces, sessions, and outcomes. Level one: spans A span is one single small unit of work inside an agent run, such as one call to the model or one call to a tool. If you remember the house on the city map, a span is the single house. Span-level evaluation asks whether this one small step was done well. For example, you can ask whether the agent chose the correct tool for the job at this moment. Span-level checks are cheap and precise, and when they fail they tell you exactly where the problem is. Level two: traces A trace, which Chapter 28 describes in depth, is the complete record of one run, made up of many spans in order. If a span is a house, a trace is the whole street. Trace-level evaluation asks whether the sequence of steps made sense together. For example, did the agent take a reasonable number of steps, avoid repeating itself, and stay within its allowed tools? Level three: sessions A session is a whole conversation or working period that may include many runs. A user might talk to a support agent for twenty minutes, asking one question after another. Each question might create its own trace, but they all belong to one session. Session-level evaluation asks whether the agent stayed consistent across the conversation. For example, did it remember what the user said earlier, did it avoid contradicting itself, and did the user become more frustrated or less frustrated as the session went on? Level four: outcomes An outcome is the final result in the world: did the user's real goal get achieved? Outcome-level evaluation ignores the path and asks only whether the thing that mattered happened. Was the flight booked on the right date? Was the bug fixed? Was the question answered correctly?