BENCHMARKS AND LEADERBOARDS

LEARNING GOALS By the end of this chapter you will be able to explain what a standard benchmark is, describe the Language Model Evaluation Harness, recognize agent-specific benchmarks and holistic leaderboards, and explain the limits of benchmarks including the danger of teaching to the test. WHAT IS A STANDARD BENCHMARK? A benchmark is a standard test that many people agree to use so that results can be compared. The word comes from surveying and measuring, where a benchmark was a marked reference point. In AI, a standard benchmark is a shared public test, meaning a fixed set of questions or tasks with agreed rules for scoring, that anyone can run on their own model or agent. Imagine a national standardized exam for students. Thousands of students take the same exam, scored the same way, and so you can say that one school scored higher than another. Without a shared exam, each school could only describe its students as pretty good, and nobody could compare. A benchmark does the same job for AI systems.