TESTING THE HARNESS ITSELF

EXAMPLES Example 1: A unit test for send_email The harness has a tool called send_email. A unit test calls it with the address "not-an-email" and checks that it returns an error saying the address is invalid. The test uses a fake mail service, and a second check confirms that the fake service received zero send requests. So the test proves the tool rejects a malformed address without ever sending a real email. A companion test uses a valid address and confirms that exactly one request reached the fake service. Both tests run in milliseconds and never bother a real person. Example 2: An integration test of the loop The team writes a scripted fake model that, for a task to look up an order, first answers with a request to call the lookup tool, and after seeing a result, answers with a final reply. The lookup tool is also mocked to return a prepared order record. The test runs the full loop and checks four things: the lookup tool was called exactly once, with the right order number; the tool result was added to the context; the loop stopped after the final reply; and the final reply contains the order status. If any hand-off between components is broken, such as the parser failing to read the tool name, the test fails, even though each component's own unit tests pass. Example 3: Regression testing The team plans to ship a new version of the context builder that compresses older messages to save space. Before shipping, they re-run last month's full eval suite, which contains 300 cases. Last month's baseline was 262 passes. The new version scores 241, and the failures cluster in long conversations where compressed messages dropped a user's special instruction. Without this regression check, the problem would have reached customers, and the team would have noticed only through complaints. They fix the compression to preserve user instructions, re-run, reach 265, and ship. Example 4: Continuous evaluation in the pipeline Every pull request that touches harness code automatically triggers the core evaluation suite of 60 cases. The agreed threshold says that the score must not fall more than three percentage points below the main branch baseline. One developer's change lowers the score by seven points, and the pipeline blocks the merge and posts a report listing the failing cases. Another developer's change lowers the score by four points, but the report shows that the failures are cases where the agent now refuses to run a dangerous shell command that the old golden answer incorrectly expected it to run. The reviewer follows the override process, corrects the golden answers, records the reason, and approves the merge.