DESIGNING EVALUATIONS (EVALS)

WHY BOTH MATTER Imagine a delivery driver who always delivers the package to the right house. The task success metric says one hundred percent. But a step-level metric shows that the driver runs red lights on the way. The first number would never reveal the second problem. In agent terms, a run can succeed at the task while using a forbidden tool, wasting a huge amount of money on repeated calls, or exposing private data in a log. Conversely, a run can fail the task even though ninety percent of its steps were right, and step-level metrics let you see that you are close and exactly where it went wrong. Use task success metrics to decide whether the agent is good enough, and use step-level metrics to figure out why it is not, and to catch risky shortcuts. PROBING SPECIFIC FAILURE MODES ON PURPOSE A failure mode is a particular, recognizable way that a system can go wrong. Examples include making up facts that are not true, which is often called hallucination; obeying hidden malicious instructions, which you will study in Chapter 38; getting stuck in a loop; or calling a tool with the wrong arguments. Probing means deliberately testing for a failure mode rather than waiting to stumble across it. A fire drill is a good comparison. A building does not wait for a real fire to find out whether people know the exit route. It stages a drill on purpose. In the same way, you design test cases that are built to trigger a particular suspected weakness.