TESTING THE HARNESS ITSELF

CONTINUOUS EVALUATION IN THE DEVELOPMENT AND RELEASE PIPELINE A pipeline is an automated assembly line that takes code from a developer's change through checks to release. Continuous integration (CI) is the practice of automatically running checks every time someone proposes a change to the code. Continuous delivery or deployment (CD) extends this by automating the steps that prepare and release the software. Continuous evaluation means building the evaluation suite into this pipeline so that measuring quality happens automatically, not just when someone remembers. HOW A TYPICAL GATE WORKS A pull request is a proposal to merge a developer's changes into the main body of code, and teammates and automated checks review it. A gate is an automatic checkpoint that a change must pass. A typical pipeline runs on every pull request that changes harness code. First, the unit tests run, which finish in seconds. Second, the integration tests run with scripted models and mocked tools. Third, a core evaluation suite runs, perhaps a few dozen to a few hundred cases. Fourth, the scores are compared with the baseline. Fifth, if the scores drop below an agreed threshold, the pipeline blocks the merge.