LLM Benchmarks Explained 2026: MMLU, GPQA, SWE-bench and How to Read Scores

Model releases quote MMLU 90%, GPQA 85% - but what do these benchmarks actually test, and how much should you trust the numbers?

💡 What You Will Learn

Model releases quote MMLU 90%, GPQA 85% - but what do these benchmarks actually test, and how much should you trust the numbers?

📜 Table of Contents

• Benchmarks Are a Language, Learn the Dialects
• The Big Four
• The Math Behind the Magic
• Where the Numbers Lie
• Bottom Line

Benchmarks Are a Language, Learn the Dialects

Every model release ships with a block of benchmark scores. Here is what the main ones test, what the numbers mean, and where they lie.

The Big Four

• MMLU - multitask knowledge across 57 subjects (STEM, humanities, social science). A very broad trivia + reasoning exam. Saturated: top models sit at 85-90%+, and the gap between #1 and #10 is a rounding error.
• MMLU-Pro - a harder, 10-option version that reduces guessing and adds more reasoning. This is the one to watch for real differentiation now.
• GPQA - graduate-level Google-proof questions (PhD-level science). Questions are so hard that experts score ~65% without web access. A model scoring 85% here is genuinely doing something.
• SWE-bench (5,599 stars) - real GitHub issues; models must produce a patch that passes the repository tests. The gold standard for coding agents. The Verified subset removes ambiguous issues. This benchmark moved the industry: coding agents are now marketed against it.

The Math Behind the Magic

Benchmarks use multiple-choice or generation-with-grading. Multiple choice is cheap but guessable (hence MMLU-Pro 10 options). Generation tasks need a judge (LLM or tests) - SWE-bench uses actual tests, which is why it is trusted. LLM-as-judge is convenient but has known biases: prefers longer answers, matches its own style.

Where the Numbers Lie

1. Leakage - if benchmark questions appear in training data, scores inflate. A constant arms race.
2. Harness variance - same model, different evaluation framework, different score. Always check the harness version.
3. Selective reporting - companies report the benchmarks they win. Absent benchmarks often mean poor results.
4. Saturation - when everyone scores 90%, the benchmark stops differentiating. GPQA and SWE-bench exist precisely because MMLU stopped being useful.

Bottom Line

Benchmark scores measure capability ceilings under exam conditions, not usefulness in your workflow. Two models within 2 points are effectively equal - choose on price, speed, context length, and your own test prompts.