THE EVALUATION HARNESS

EXAMPLES Example 1: Span-level evaluation Imagine a travel agent that has three tools: search_flights, search_hotels, and send_email. A user writes, "Find me flights from Boston to Denver next Friday." At the first tool-calling step, the agent calls search_hotels. That is the wrong tool, even if the agent later corrects itself. A span-level check reads the span that records this tool call and applies a simple rule: given a request that mentions flights, the first tool chosen should be search_flights. The check fails, and the report points straight to this one step. Notice how helpful this is for debugging. Instead of reading a long transcript hunting for the problem, you are told exactly which step went wrong and what the correct choice would have been. Example 2: Outcome-level evaluation Now imagine the same travel agent, and this time we only care about the final result. The test case says that the user wants a flight booked from Boston to Denver on a specific Friday, and the correct final state is a booking record with that route and that date. After the agent finishes, the evaluation harness looks at the booking system, not at the conversation. If the record exists with the right route and date, the outcome check passes. It does not matter whether the agent used four steps or nine, or whether it phrased its replies beautifully. Outcome-level scoring answers the question that the user cares about most, which is whether the goal was achieved. Its limit is that it says little about why a failure happened, and it can award a pass to a run that succeeded through a risky route. Example 3: Offline versus online A company has a customer-support agent, and the engineers have changed how the harness builds the context shown to the model. Before release, they run the offline evaluation: a fixed set of 200 test tasks with known correct resolutions. The old version scored 172 out of 200, and the new version scores 176 out of 200. That looks like an improvement, so they release it. After release, the online evaluation scores a random sample of live conversations, perhaps one in twenty, using an automatic judge. Within two days the online scores for conversations about refund disputes in Spanish are noticeably lower than before. None of the 200 offline tasks were in Spanish, so the dress rehearsal missed this. The team uses the rollback rule they wrote earlier, or fixes the problem quickly, and then adds Spanish refund disputes to the offline set so the gap is closed. This is exactly the partnership described above. Example 4: A curated dataset used repeatedly A support team collects 50 tricky customer tickets. Each is a case where earlier versions of the agent made a mistake, such as a customer who writes two questions in one message or a ticket that refers to an order number that does not exist. An experienced support person writes the known-correct resolution for each ticket. Every time the engineers change the harness, they run these same 50 tickets. Because the cases never change, the result is a fair, apples-to-apples comparison. If a change moves the score from 38 out of 50 to 44 out of 50, it genuinely helped on the hard cases. If it moves from 38 down to 31, the team knows right away that the change hurt, even if the agent still looks fine on easy tickets.