DESIGNING EVALUATIONS (EVALS)

THE ANATOMY OF A SINGLE EVAL CASE One eval case, sometimes called a test case, usually contains the following parts. First, an identifier, such as a number, so you can refer to the case later. Second, the input: the question, document, or request given to the agent. Third, any setup the case needs, such as which files exist or what the fake database contains. Fourth, the expected result, often called the golden reference answer, which we explain shortly. Fifth, a case type label, such as normal, edge, or adversarial. Sixth, a scoring rule that says how to compare the agent's result to the expected result.