LLM Evaluation Metrics for RAG 2026: Faithfulness, Relevancy and Context Precision
Your RAG system answers feel good, but how do you measure whether retrieval actually helped? Classic metrics like accuracy miss what RAG does wrong. Here are the metrics that matter.
💡 What You Will Learn
Your RAG system answers feel good, but how do you measure whether retrieval actually helped? Classic metrics like accuracy miss what RAG does wrong. Here are the metrics that matter.
📜 Table of Contents
Why Ordinary Metrics Fail RAG
Accuracy on a fixed test set tells you the model answered, not why. RAG failures come in two flavors: retrieval returned the wrong context, or the model ignored good context. You need metrics that isolate these two stages.
The Four RAG Metrics That Matter (2026)
- Faithfulness - is every claim in the answer supported by the retrieved context? This catches hallucination: the model inventing facts not in the documents. Score per-claim: each claim is checked against the context, faithfulness = supported claims / total claims.
- Answer relevancy - does the answer actually address the question? An answer can be faithful to context yet irrelevant to what was asked.
- Context precision - of the chunks retrieved, how many were actually needed? Measures retrieval noise: if 8 chunks come back and only 2 matter, precision is low even when the answer is right.
- Context recall - did retrieval find all the chunks needed to answer? Missed a key paragraph and the answer misses a key fact.
The RAGAS Framework
RAGAS (Retrieval Augmented Generation Assessment) is the standard open source tool for these metrics, and it introduced the concept of LLM-based metric scoring: a judge LLM scores each dimension on generated answers. It works without ground-truth labels for faithfulness and relevancy, which is why teams adopt it - you can evaluate the day you ship.
The Triad You Should Always Track
Faithfulness + answer relevancy + context precision form the evaluation triad. Dashboards like Arize Phoenix (10k stars) and Langfuse ship these as built-in evals, so you get them in your existing traces instead of a separate offline script.
A Practical Threshold Setup
- Faithfulness below 0.8: your model ignores or misuses context - fix prompting or the retriever
- Context precision below 0.5: retrieval is noisy - tighten chunking, add reranking
- Answer relevancy below 0.7: question understanding is broken - check query rewriting
The Two-Track Testing Pattern
Track 1 (offline): a golden set of 100-200 question-answer pairs, scored weekly. Track 2 (online): sample 50 real production queries per week, score with a judge LLM. The offline set catches regressions; the online set catches drift in real user questions.
