TESTING THE HARNESS ITSELF

LEARNING GOALS By the end of this chapter you will be able to explain unit testing of tools and components, integration testing of the full loop, regression testing, evaluation-driven iteration, and continuous evaluation in the development and release pipeline. A DIFFERENT KIND OF TESTING So far in this part, we have measured how good an agent is at tasks. Now consider that the harness is also ordinary software. It contains code that builds context, parses model output, checks permissions, calls tools, and saves state. Ordinary software can have ordinary bugs, such as a typo, a crash, or a wrong calculation, that have nothing to do with the model's intelligence. Imagine a car. Road tests tell you how the car performs while driving, but the factory also tests the brakes, the engine, and the electrical system separately on a workbench. This chapter is about testing the parts and the whole of the harness as software, and about wiring all that testing into the way you build and release. UNIT TESTING OF TOOLS AND COMPONENTS A unit is a small piece of code that does one job, such as a function that validates an email address. A unit test is a small automatic check that runs one unit with a chosen input and verifies that the output is what you expect. Unit tests are fast, usually running in a fraction of a second, and precise, since a failure points to one small piece of code.