Tool School: Benchmarking 101 (How To Read AI Model Report Cards)
Authors: Hannah Stulberg and Akshat Khandelwal | Published: 2026-03-31
Learn to read the scorecard behind every AI model launch. Core thesis: “The best model is the one that does the work you need, at a price you can justify, on benchmarks you understand.”
Benchmarks 101
Analogy: Benchmark scores are like SAT scores – a high SAT score doesn’t make someone a great colleague. Each measures one narrow skill.
Three Scoring Systems
| System | How it works | Notes |
|---|---|---|
| Pass@1 | One attempt per question | Closest to actual user experience. Most commonly reported |
| Pass@k | Multiple attempts (8 or 16); any correct = pass | Always higher than pass@1. Some companies report this without clear labeling |
| Elo rating | Head-to-head ranking (like chess) | Chatbot Arena uses this. Measures preference, not correctness |
Four Benchmark Categories
- Knowledge & Reasoning – academic/professional knowledge (MMLU, GPQA Diamond, HLE, GDPval)
- Coding – writing working code (SWE-bench Verified, SWE-bench Pro, SWE-Lancer)
- Reasoning – novel problem-solving (ARC-AGI-2, ARC-AGI-3)
- Agentic & Computer Use – sustained multi-step professional work (OSWorld, tau2-bench, BigLaw Bench, Terminal-Bench 2.0)
Key Benchmarks
| Benchmark | What it tests | Current status |
|---|---|---|
| MMLU | 16K questions, 57 subjects | Saturated (88-93%) |
| GPQA Diamond | 448 PhD-level science questions | Approaching saturation (91-94%) |
| HLE | 2,500 expert questions, 100+ subjects (published in Nature) | Still differentiating (40-45%) |
| SWE-bench Verified | 500 real GitHub issues | Contaminated – every frontier model trained on solutions |
| SWE-bench Pro | 1,900 tasks with private test cases | Current best coding benchmark |
| SWE-Lancer | 1,400+ real Upwork tasks ($50 to $32K) | 80% SWE-bench = only 26% SWE-Lancer |
| ARC-AGI-3 | Interactive game environments, no instructions | Widest human-AI gap (humans 100%, best AI 0.37%) |
| ARC-AGI-2 | Visual pattern puzzles | Scores still spread (53-77%). Measures cost alongside accuracy |
| OSWorld | Real computer tasks in OS environments | GPT-5.4 surpassed human performance (75% vs 72.4%) |
Why Benchmarks Expire
- Saturation – scores converge, spread disappears (MMLU lasted ~3 years, GPQA ~2)
- Contamination – test answers leak into training data (SWE-bench Verified confirmed contaminated)
- Goodhart’s Law – when a measure becomes a target, it ceases to be a good measure
Three Questions for Any Benchmark
- How long has the test been public?
- Who created it?
- Are the scores still spread out?
Trust Tiers
- High relevance: ARC-AGI-3, ARC-AGI-2, HLE, SWE-bench Pro, OSWorld, Terminal-Bench 2.0
- Moderate: GPQA Diamond, tau2-bench, BigLaw Bench, GDPval, MMLU Pro
- Low: MMLU, SWE-bench Verified, HumanEval
Formula: independent creation + wide score spread + contamination resistance = high relevance
Head-to-Head (as of article date)
| Strength | Leader |
|---|---|
| Academic reasoning | Gemini 3.1 Pro |
| Professional tasks | GPT-5.4 |
| Enterprise agentic work | Opus 4.6 |
Top 5 models separated by just 19 Elo points in Chatbot Arena.
Cost
- API pricing: input tokens vs output tokens, per million
- Same benchmark scores, wildly different prices: DeepSeek R1 at $2.19/M output vs $15 (GPT-5.4) vs $25 (Opus 4.6)
- Sonnet 4.6 delivers ~95% of Opus 4.6 performance at 60% of the cost
- For consumer subscriptions ($20/month), cost shows up as rate limits, not per-token
Four Things to Watch in Any Launch
- What benchmarks are missing from the scorecard? (selective disclosure)
- What baseline is the company comparing against? (may be outdated)
- How does Chatbot Arena compare to the benchmark story?
- How transparent is the methodology?
Independent Sources for Checking Scores
- Chatbot Arena, Artificial Analysis, ARC Prize Leaderboard, Scale AI SWE-bench Pro Leaderboard, Stanford CRFM Foundation Model Transparency Index