BENCHMARKS AND LEADERBOARDS

EXAMPLES Example 1: A shared scoreboard Two companies each claim their model is better at answering science questions. Without a benchmark, each cites impressive-looking sample answers. With a standard benchmark, both models answer the same 1,000 fixed questions under the same rules. Model A gets 812 correct and Model B gets 790 correct. Now there is a shared, repeatable comparison. A careful reader still asks whether the questions leaked into training, and whether a 22-question gap is large enough to be meaningful, but the comparison is far better than impressions. Example 2: The Language Model Evaluation Harness in use A research group wants to compare three of its models on a dozen benchmark suites. Rather than writing separate custom code for each suite, they use the Language Model Evaluation Harness. They list the models and the suites, run one command, and receive a table of scores produced with the same formatting and scoring rules for every model. A second group at another university repeats the run and gets nearly the same table, which makes the two groups' reports comparable. Example 3: An agent-specific benchmark A team building a coding agent wants to know if it can fix real bugs. A question-answering benchmark would reveal only what the model knows. An agent benchmark such as SWE-bench instead gives the agent a real project and a real bug report, lets it read files, run commands, and edit code, and then runs the project's tests to check whether the bug is truly fixed. The score measures multi-step tool use and task completion, the very abilities that matter for an agent, not just recall. Example 4: Teaching to the test A team uses one fixed set of 300 public benchmark questions as its only measure. Over six releases, the score climbs from 61 to 83 percent. During the same period, customer-satisfaction ratings stay flat, and support escalations rise slightly. When engineers investigate, they find that several changes added instructions tailored to the style of those exact questions, which helped the benchmark and did nothing for real users. The team introduces a hidden held-out set and a monthly refresh of eval cases from real tickets. The next release shows a smaller but honest improvement on both the hidden set and satisfaction.