WHY HARNESS ENGINEERING EMERGED AS ITS OWN FIELD

AUTOMATED HARNESS OPTIMIZATION Automated harness optimization means tuning the harness, rather than the model, with the help of automated experiments. The idea has a simple logic. If the harness matters so much, then improving the harness is a major way to improve the system. And if you have a good set of test tasks with scores, you can try many harness variations automatically and keep the best one. A harness has many adjustable choices. How many times should a failed step be retried? How many tokens of history should be kept before older material is summarized? Which tools should be visible to the agent at the start? How should the instructions be phrased and ordered? How many steps are allowed? Each choice is a setting, sometimes called a configuration, and different combinations yield different results. In automated optimization, a pipeline, which is an automated series of steps, runs the agent on the same fixed set of tasks under several configurations, records scores, costs, and times for each, compares them, and promotes the best one. Researchers have also explored letting a model itself propose improvements to a harness, such as suggesting a better instruction or a new tool description, and then testing those suggestions automatically. This is an active area of research, and results are early, so treat specific claims about it with caution. What is solid is the underlying method. Change one thing at a time, measure on the same tasks, compare fairly, and decide by a rule fixed in advance. This method works whether the experimenter is a person with a spreadsheet or an automated system. Watch for common mistakes. One is testing on too few tasks, so that a difference is just luck. Another is changing several settings together and then not knowing which one mattered. A third is tuning the harness to do well on the test tasks so narrowly that it does worse on new, real tasks, a trap called overfitting, which means fitting the test so closely that you lose the ability to generalize. A fourth is looking only at success rate and ignoring cost and speed. You will study evaluation methods in depth in Part Eight and the economics in Part Thirteen.