LLM Benchmarks Explained 2026: MMLU, GPQA, SWE-bench and How to Read Scores

Model releases quote MMLU 90%, GPQA 85% - but what do these benchmarks actually test, and how much should you trust the numbers?

💡 What You Will Learn

Model releases quote MMLU 90%, GPQA 85% - but what do these benchmarks actually test, and how much should you trust the numbers?

## Benchmarks Are a Language, Learn the Dialects Every model release ships with a block of benchmark scores. Here is what the main ones test, what the numbers mean, and where they lie. ## The Big Four - **MMLU** - multitask knowledge across 57 subjects (STEM, humanities, social science). A very broad trivia + reasoning exam. Saturated: top models sit at 85-90%+, and the gap between #1 and #10 is a rounding error. - **MMLU-Pro** - a harder, 10-option version that reduces guessing and adds more reasoning. This is the one to watch for real differentiation now. - **GPQA** - graduate-level Google-proof questions (PhD-level science). Questions are so hard that experts score ~65% without web access. A model scoring 85% here is genuinely doing something. - **SWE-bench (5,599 stars)** - real GitHub issues; models must produce a patch that passes the repository tests. The gold standard for coding agents. The Verified subset removes ambiguous issues. This benchmark moved the industry: coding agents are now marketed against it. ## The Math Behind the Magic Benchmarks use multiple-choice or generation-with-grading. Multiple choice is cheap but guessable (hence MMLU-Pro 10 options). Generation tasks need a judge (LLM or tests) - SWE-bench uses actual tests, which is why it is trusted. LLM-as-judge is convenient but has known biases: prefers longer answers, matches its own style. ## Where the Numbers Lie 1. **Leakage** - if benchmark questions appear in training data, scores inflate. A constant arms race. 2. **Harness variance** - same model, different evaluation framework, different score. Always check the harness version. 3. **Selective reporting** - companies report the benchmarks they win. Absent benchmarks often mean poor results. 4. **Saturation** - when everyone scores 90%, the benchmark stops differentiating. GPQA and SWE-bench exist precisely because MMLU stopped being useful. ## Bottom Line Benchmark scores measure capability ceilings under exam conditions, not usefulness in your workflow. Two models within 2 points are effectively equal - choose on price, speed, context length, and your own test prompts.