THE OPEN-SOURCE LANDSCAPE

COMMUNITY BENCHMARKS AND SHARED DATASETS A benchmark is a standard set of tasks, with a standard way of scoring, used to compare different systems fairly. A dataset is a collection of example data, such as realistic tasks with known correct answers. Imagine many students sitting the same exam under the same rules. Their scores can be compared directly, which would be impossible if each student took a different exam. A community benchmark does this for agents. Many teams run their own harnesses against the same shared tasks, and publish their scores. This gives you an apples-to-apples comparison between unrelated projects, and it helps you see where your harness stands. Benchmarks also have well-known weaknesses, which you should keep in mind, building on the evaluation lessons of Part Eight. Over-tuning: if you keep changing your harness until the benchmark score rises, you may be teaching it to pass that particular exam, not to do real work. This is related to a general warning often phrased as follows: when a measure becomes a target, it stops being a good measure. Leakage: if the benchmark tasks or answers appear in the data a model learned from, the model may have effectively seen the answers beforehand, and the score overstates its ability. Mismatch: tasks in a public benchmark may look nothing like your real users' tasks. Staleness: as models improve, an old benchmark may become too easy, and scores bunch together near the top. Setup differences: scores can change with details such as the tools provided, the time allowed, and the number of attempts, so compare only results gathered under the same conditions. The healthy way to use a benchmark is as one signal among several. Combine it with your own evaluation set built from your own real tasks, as taught in Chapters 32 and 34, and keep some tasks private so you can check for over-tuning.