LLM Evaluation Metrics Papers in 2026
The MMLU, HumanEval and GSM8K scores on today's model cards, the faithfulness and context-precision metrics in RAG projects, and LLM-as-judge in CI all trace back to a handful of foundational papers. Reading them tells you what each score measures and what it hides.
💡 What You Will Learn
The MMLU, HumanEval and GSM8K scores on today's model cards, the faithfulness and context-precision metrics in RAG projects, and LLM-as-judge in CI all trace back to a handful of foundational papers.
📜 Table of Contents
LLM Evaluation Metrics Papers: 8 Foundations
The MMLU, HumanEval and GSM8K numbers on model cards, the faithfulness metrics in RAG projects and LLM-as-judge in CI all trace back to a handful of papers. Reading them tells you what each score measures and what it hides.
1. The eight
| Paper | Year | Problem it solved | What it left behind |
|---|---|---|---|
| MMLU | 2020 | quantifying general knowledge | a 57-subject multiple-choice benchmark |
| HELM | 2022 | single-metric evaluation is too narrow | multi-scenario, multi-metric standardized reporting |
| GSM8K | 2021 | measuring math reasoning | grade-school word problems as a reasoning standard |
| HumanEval | 2021 | measuring code ability | handwritten problems plus pass@k |
| TruthfulQA | 2021 | does the model confidently make things up | a misconception-focused truthfulness benchmark |
| Chain-of-Thought | 2022 | can prompting unlock reasoning | step-by-step reasoning before the answer |
| RAGAS | 2023 | how to evaluate RAG | faithfulness, answer relevancy, context precision and recall |
| MT-Bench / LLM-as-a-Judge | 2023 | human eval is slow and costly | strong models as judges, with judge bias studied |
All are free on arXiv. Check the original pages for exact titles, years and authors.
2. What each one is for
- MMLU made general capability comparable - and is now saturated, so treat it as a floor, not a differentiator
- HELM's real contribution is methodology: many scenarios, many metrics, full disclosure
- GSM8K separates "sounds right" from "computes right"
- HumanEval introduced pass@k - one success among k samples - and later code benchmarks follow it with harder, more repository-like tasks
- TruthfulQA asks about common misconceptions to see whether a model follows the user's error or corrects it
- Chain-of-Thought showed that asking for steps before the answer raises accuracy; today's reasoning models train that behavior in
- RAGAS splits failures into retrieval (context precision, context recall) and generation (faithfulness, answer relevancy) so you know where to fix
- MT-Bench / LLM-as-a-Judge validated model judges and honestly documented their biases: position, verbosity, and self-preference
3. Two useful additions
BIG-bench (2022) for breadth checks, and SWE-bench (2023) - real repository bug fixes, the arena for coding agents in 2026.
4. How to use them in practice
| Goal | Use | Watch out |
|---|---|---|
| Base model selection | general + math + code benchmarks | a filter, not a verdict |
| RAG systems | faithfulness, context precision and recall | inspect retrieval and generation separately |
| Release regression | your own eval set plus an LLM judge | audit the judge, bias compounds |
| Coding agents | repository-level tasks | contamination is worst here; private sets are more trustworthy |
Public benchmarks saturate and risk contamination. What decides a release is your own eval set, drawn from real business questions with reference answers and maintained across versions. Public scores compare; your set decides.
FAQ
Where to start? Applications: RAGAS and LLM-as-a-Judge. Model selection: MMLU, GSM8K, HumanEval. Methodology: HELM.
Do high scores mean it works for me? No. Benchmarks measure specific tasks; your domain, prompts and context length differ.
Are model judges reliable? Reasonably so for pairwise comparisons with fixed rubrics, but they prefer certain positions, longer answers and their own outputs. Swap order and re-run, and calibrate with human sampling.
Anything directly reusable? RAGAS and LLM-as-a-Judge have open implementations, and the MMLU, GSM8K and HumanEval datasets are public - check each license.
Note: a reading guide only; verify titles, years, authors and dataset sizes on arXiv and the official repos.
❓ FAQ
Do I need to read all eight?
Start with MMLU, RAGAS, and LLM-as-a-Judge - they cover 80% of daily practice.
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only โ no paid placements.