THE EVALUATION HARNESS

THE FOUR MAIN PIECES OF AN EVALUATION HARNESS First, there are test cases, which are the situations you feed to the agent, such as a question, a document, or a request. Second, there is a runner, which is the code that actually gives each test case to the agent and records what happens. Third, there are scorers, which are the rules or judges that decide how good each result is. Fourth, there is a report, which collects all the scores in one place so that you can compare this version of your harness with the last version. Notice a subtle point. An evaluation harness measures the whole agent system, including the model, the scaffold, and the harness around it. You are not only asking whether the model is smart. You are asking whether the complete machine does its job.