THE EVALUATION HARNESS

INPUTS FROM CURATED DATASETS A curated dataset is a collection of test cases that people deliberately chose, wrote, or cleaned up, usually with known correct answers attached. Curated means carefully selected and maintained, the way a museum curator selects pieces for an exhibit. The strength of a curated dataset is control. You can include exactly the tricky situations you care about, you can keep the set stable so scores are comparable from month to month, and you can store the correct answer beside each case. The weakness is that a curated set can drift away from what real users really do. A mature team uses both. They keep a stable curated set for repeatable comparisons and regularly add new cases mined from production, especially cases where the agent failed, so that the same mistake is never allowed to happen twice unnoticed. OFFLINE VERSUS ONLINE EVALUATION The last big idea of this chapter concerns timing.