BENCHMARKS AND LEADERBOARDS

CHECK YOUR UNDERSTANDING • Why do benchmarks help compare models? ◦ Answer: Because every model faces the same fixed questions and scoring rules, giving a repeatable result instead of impressions. • How do agent benchmarks differ from older question-answering benchmarks? ◦ Answer: They test multi-step tool use and task completion in realistic workflows rather than single-turn knowledge recall. • What is contamination? ◦ Answer: When benchmark questions or answers leak into the data a model was trained on, making scores misleading. • Describe one safeguard against teaching to the test. ◦ Answer: Keep a hidden held-out test set that is never tuned against, or refresh eval cases regularly from production. • Why should you not rely only on a public leaderboard? ◦ Answer: Because it may not reflect your own tasks, tools, and users.