DESIGNING EVALUATIONS (EVALS)

CHOOSING HOW MANY CASES AND WHAT MIX A beginner often asks how many cases are enough. There is no single magic number, but there are useful guidelines. Start small and real. Twenty to fifty well-chosen cases that you understand deeply are more useful than a thousand sloppy ones. Then grow the dataset over time, adding a new case every time you discover a new kind of failure. A very helpful way to balance a dataset is to include three kinds of cases. Normal cases are everyday, typical requests that the agent should handle easily. Edge cases are ambiguous or unusual requests, such as a resume that is only two lines long or a question that could be read two ways. Adversarial cases are deliberately tricky or malicious inputs, such as an attempt to trick the agent into breaking its rules. A dataset of 100 cases might hold 70 normal cases, 20 edge cases, and 10 adversarial cases. The proportion is a design choice. The normal cases tell you whether the everyday experience is good, the edge cases tell you whether the agent behaves sensibly when things get strange, and the adversarial cases tell you whether the agent resists abuse.