TESTING THE HARNESS ITSELF

PRACTICAL DESIGN QUESTIONS How big should the per-change suite be? Real model calls cost money and time, so teams often run a small, fast subset on every change and a larger full suite nightly or before release. How should randomness be handled? Run borderline cases multiple times, or allow a tolerance band, so that a harmless wobble does not block good changes. What about legitimate drops? Sometimes a score falls for a good reason. For example, you fix an unsafe shortcut that previously inflated scores, and the agent now correctly refuses some requests that the old golden answers treated as acceptable. A mature gate therefore includes an override process: a named person reviews the cause, records the justification, updates the affected eval cases if needed, and approves the change explicitly. The override should be rare and visible, never a casual click. Also store results over time so you can see trends, and make failures easy to read, since an engineer must quickly understand which case failed and why.