LLM Evaluation Metrics for RAG 2026
Your RAG answers feel fine, but how do you prove retrieval helped? Accuracy cannot tell you which stage failed - you need metrics that separate retrieval from generation.
📜 Table of Contents
LLM Evaluation Metrics for RAG 2026: Faithfulness, Relevancy and Context Precision
Accuracy on a fixed test set tells you whether an answer was right, not why. RAG fails in two orthogonal ways: retrieval returned the wrong context, or the model ignored the right context. Those need different metrics. Mix them together and you never know whether to fix retrieval or the prompt.
Why accuracy is not enough
A RAG system has two stages. Retrieval failure means the needed document was never found, or was buried in noise - no model can answer from that. Generation failure means the answer was in context but the model invented something else.
Accuracy collapses both into one number. So evaluate in layers: score retrieval separately from generation.
Four core metrics
| Metric | Measures | Catches | Target |
|---|---|---|---|
| Faithfulness | Whether each claim in the answer is supported by retrieved context | Hallucination | 0.8+ |
| Answer Relevancy | Whether the answer addresses the question | Evasive, off-topic answers | 0.7+ |
| Context Precision | Share of retrieved chunks actually used | Retrieval noise | 0.5+ |
| Context Recall | Whether all needed information was retrieved | Missed documents | 0.7+ |
Faithfulness matters most. Break the answer into claims and check each against context. An answer can be fully faithful yet irrelevant, or sound right with zero support.
Answer Relevancy works by reverse generation: ask a model what question the answer addresses, then compare to the real question. Rambling answers fail here.
Context Precision reflects ranking quality - useful chunks should come first. Noise at the top is what misleads the generator.
Context Recall needs ground truth and measures what was missed. It is the most annotation-hungry metric.
RAGAS: replace human raters with a judge model
RAGAS scores each sample along these dimensions using a strong model as judge, so no hand labeling is needed for the first three.
pip install ragas datasets
from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (faithfulness, answer_relevancy,
context_precision, context_recall)
data = Dataset.from_dict({
"question": ["What is the return policy window?"],
"answer": ["Returns accepted within 30 days, packaging intact."],
"contexts": [["We accept returns within 30 days; packaging must be intact."]],
"ground_truth": ["30 days, packaging intact."],
})
print(evaluate(data, metrics=[faithfulness, answer_relevancy,
context_precision, context_recall]))
Swap in your system's real outputs and run 100-200 samples to localize the failure. The judge model needs real capability; a weak judge biases everything.
Thresholds and what to fix first
| Symptom | Likely cause | Fix first |
|---|---|---|
| Faithfulness < 0.8 | Model improvising beyond context | Tighten prompt: answer only from provided material |
| Context precision < 0.5 | Retrieval too noisy | Smaller chunks, add reranking, lower top-k |
| Recall < 0.7 | Needed content missing | Chunking strategy, hybrid keyword + vector search |
| Relevancy < 0.7 | Question misread | Query rewriting, keep conversation context |
Change one layer at a time. Fixing chunking when faithfulness is low is work in the wrong place.
Two tracks: offline and online
Offline: keep a 100-200 item golden set covering typical and edge questions; run before and after every change as a regression net.
Online: sample 50 real queries weekly and score them. Production distributions never match your test set.
You need both. Offline alone drifts from reality; online alone cannot detect regressions.
Common traps
Judge bias: using the same model to generate and score favors its own style. Use a different judge.
Oversized context: dumping whole documents in front of the judge inflates precision with irrelevant-but-real quotes. Score the context that actually reached the prompt.
Test set leakage: golden questions appearing in few-shot examples inflate scores, then collapse in production.
Single-metric optimization: pushing faithfulness to 0.95 while relevancy drops to 0.5 gives you a system that only parrots context. Read all four together.
Starting without ground truth
Run the three metrics that need no reference answers first - faithfulness, answer relevancy, context precision - which already localize most problems. Generate candidate ground truths with a strong model and hand-correct 20%, far cheaper than full annotation.
FAQ
Must RAGAS use OpenAI models? No. Local or other API models work; stronger judges are more reliable but cost more.
Scores swing wildly? Increase sample size and average runs. Under 100 samples, noise alone can flip conclusions.
Does fine-tuning need different evaluation? Yes - task metrics like accuracy and format compliance, not these retrieval-stage metrics.
Evaluation is expensive? Run the full set offline, sample online, and use a cheap model to triage before escalating to a strong judge.
Note: RAGAS metric names and interfaces change between versions; check official docs. Thresholds are heuristics to tune for your domain.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.