DESIGNING EVALUATIONS (EVALS)

TASK SUCCESS METRICS A task success metric asks whether the whole job was done correctly. Examples include whether the agent booked the correct flight, whether the code it wrote passes its tests, and whether the answer it gave matches the golden answer. Task success metrics are what the business and the user ultimately care about. They are usually easy to explain to a non-technical person. Their weakness is that a failure tells you very little about where the problem lives. Step-level metrics A step-level metric examines something that happened during the run. For example, did the agent use the correct departure city when it searched at step two? Did it stay under the allowed number of tool calls? Did it ask for approval before a risky action? Did it ever read a file it was not supposed to read? Step-level metrics are diagnostic, meaning they help you find the cause of a problem. They also catch bad behavior that task success hides.