BENCHMARKS AND LEADERBOARDS

CHAPTER SUMMARY Standard benchmarks are shared public tests that make results comparable across models and teams. The Language Model Evaluation Harness is an open-source framework that runs many benchmark suites in a standardized way. Agent-specific benchmarks test multi-step tool use and task completion, and holistic leaderboards weigh many measures at once. Benchmarks have limits, including mismatch with your work, contamination, saturation, narrow metrics, and teaching to the test, so they should complement and never replace your own evals. KEY TERMS REVIEW Benchmark: a standard shared test used for comparison. Leaderboard: a public ranking table of scores. Holistic leaderboard: one that judges across many scenarios and measures. Open-source: software whose code is public. Contamination: benchmark content leaking into training data. Saturation: scores so high that the test no longer separates systems. Teaching to the test: tuning a system to score well on a known test rather than to be good generally. Held-out set: a hidden test set not used for tuning.