TESTING THE HARNESS ITSELF

REGRESSION TESTING A regression is when something that used to work stops working after a change. The word means going backward. Regression testing means checking that new changes do not break old behavior. Software changes constantly, and each change risks a side effect somewhere unexpected. A fix to the context builder might accidentally cause the agent to lose track of user instructions in long conversations. A regression suite is a collection of tests and eval cases that capture behavior you already depend on. Before shipping any change, you re-run the suite. HOW REGRESSION SUITES GROW Every time a bug is found and fixed, add a test that would have caught it. This simple habit means a bug that is fixed once stays fixed forever. For AI systems, regression testing includes the evaluation suite you built in the earlier chapters, since model-related behavior can also silently degrade. Because agent outputs vary, regression checks often compare scores against a baseline, which is the previous version's recorded score, with an allowed tolerance rather than demanding identical outputs.