BENCHMARKS AND LEADERBOARDS

PRACTICE LAB CONNECTION In the Resume Review Agent lab (Chapter 68), you built your own eval set. Now compare that eval set with the task style of a public agent benchmark. Ask what kinds of abilities the public benchmark checks that your lab set does not, such as long multi-step tasks, recovery from tool errors, or unusual input formats. Test the prompts below on that comparison. ARTIFICIAL INTELLIGENCE ASSISTANT PROMPTS [START OF PROMPT] Compare our internal eval set against a relevant public agent benchmark's task style and tell me what blind spots our internal set has: our eval set = [PASTE LAB OR PRODUCTION EVAL SET HERE], benchmark style/description = [PUBLIC BENCHMARK DESCRIPTION OR NAME]. Output format: a gap analysis table (Benchmark Capability Tested, Covered by Our Eval Set? Y/N, Suggested New Eval Item). [END OF PROMPT] [START OF PROMPT] We have been tuning our harness against this fixed internal benchmark for several release cycles: [PASTE BENCHMARK/EVAL SET AND RECENT SCORE HISTORY HERE]. Assess whether we show signs of "teaching to the test" (benchmark score rising while other quality signals stagnate or worsen), and propose two concrete safeguards to detect or prevent this going forward. Output format: an assessment paragraph plus a two-item safeguard list with implementation detail. [END OF PROMPT]