RAG Evaluation Metrics in 2026: Measure Retrieval Quality with Ragas (15k Stars)

Your RAG app answers confidently but half the citations are wrong. Classic symptom: users stop trusting it. You need metrics, not vibes.

## RAG Evaluation Metrics: The Numbers That Tell You Retrieval Is Broken Most RAG failures are retrieval failures in disguise. The generator answers fine; the retriever just handed it the wrong documents. Ragas is the open-source framework built to measure this split, and with 15,076 GitHub stars (August 2026) it is the most widely used RAG evaluation library in the ecosystem. ## The Four Metrics That Matter Ragas defines four core metrics that map to the two failure modes: **Retrieval side:** - **Context Precision** - of the chunks the retriever returned, how many were actually relevant to the question. - **Context Recall** - of all relevant chunks that exist in the corpus, how many did the retriever find. **Generation side:** - **Faithfulness** - does the answer contain claims not supported by the retrieved context? Low faithfulness means hallucination. - **Answer Relevancy** - does the answer actually address the question asked, or did it drift to something adjacent? ## A Concrete Workflow ```python from ragas import evaluate from ragas.metrics import faithfulness, context_recall result = evaluate(dataset, metrics=[faithfulness, context_recall]) ``` Feed it question-answer-context triples, and Ragas uses an LLM judge to score each. Run this on every corpus change. If context recall drops from 0.82 to 0.61 after you switched embedding models, you caught the regression in ten minutes instead of discovering it from user complaints. ## Where Teams Get Stuck The most common mistake is evaluating only faithfulness. A RAG system can score 0.9 faithfulness while retrieving the same two chunks for every question - technically faithful, practically useless. Always pair a retrieval metric with a generation metric. If you only track one number, make it context recall, because it degrades first and predicts the other metrics falling a week later. ## FAQ **Does Ragas need an LLM API?** It uses an LLM as judge, so yes, but it works with any provider including local models. **How many samples do I need?** 50-100 representative questions give a stable signal; 20 works for a first pass. **Is it only for RAG?** It also evaluates agent tool-use traces and summarization tasks. **Ragas vs plain pytest?** Ragas is purpose-built for RAG dimensions; pytest is the runner underneath.