BENCHMARKS AND LEADERBOARDS

SAFEGUARDS AGAINST TEACHING TO THE TEST First, keep a hidden or held-out test set that you never tune against and that only a few people can see, consulted rarely. Second, refresh your eval cases regularly with new examples from production. Third, track several independent quality signals, such as user satisfaction, escalation rates, and human review scores, and be suspicious when one improves while the others do not. Fourth, report results with their uncertainty, since small score differences may be noise. Fifth, always combine public benchmarks with your own task-specific evals.