BENCHMARKS AND LEADERBOARDS

AGENT-SPECIFIC BENCHMARKS Early benchmarks mostly asked single questions, such as knowledge quizzes or reading comprehension. But an agent does far more than answer a question. It plans, uses tools, takes many steps, recovers from errors, and works in an environment. Therefore agent-specific benchmarks test multi-step tool use and task completion across realistic workflows. Several well-known examples exist. Software Engineering benchmark (SWE-bench) presents real software bugs drawn from open-source projects and checks whether an agent can produce a code change that makes the project's tests pass. General AI Assistants benchmark (GAIA) tests general assistant abilities that need reasoning, web browsing, and tool use to answer questions. WebArena places agents in realistic simulated websites and asks them to complete tasks such as shopping or forum posting. tau-bench tests agents in simulated customer-service settings, where an agent must follow policies while talking to a simulated user and using tools. New agent benchmarks appear frequently, and older ones can become too easy as models improve, so always check the current status of any benchmark before relying on it. When reading about an agent benchmark, ask the following questions. What exactly is the task style? How is success decided? Does the agent run in a realistic environment? Is the test public, meaning the questions might have been seen during training?