RAGAS: Open Source LLM Evaluation Framework Tutorial for 2026

RAGAS (RAG Assessment) is the leading open source framework for evaluating RAG pipelines. With 14,855 stars on GitHub, it is the tool most AI engineers reach for. ## What RAGAS Measures ### Retrieval Metrics - Context Precision: Are top-ranked docs the most relevant? - Context Recall: Did we retrieve all relevant docs? ### Generation Metrics - Faithfulness: Does answer stay within context? - Answer Relevancy: Does answer address the question? - Answer Correctness: Is answer factually correct? ## Installation ```bash pip install ragas ``` You need a judge LLM (GPT-4 or Claude) to score answers. ```python import os os.environ["OPENAI_API_KEY"] = "your-key-here" ``` ## Creating a Dataset ```python from datasets import Dataset data = { "question": ["What is RAG?"], "answer": ["RAG stands for Retrieval-Augmented Generation..."], "contexts": [["RAG is a technique that..."]], "ground_truth": ["RAG is..."] } dataset = Dataset.from_dict(data) ``` ## Running Evaluation ```python from ragas import evaluate from ragas.metrics import faithfulness, answer_relevancy result = evaluate(dataset=dataset, metrics=[faithfulness, answer_relevancy]) print(result) ``` ## Interpreting Scores | Score | Meaning | |-------|---------| | >0.90 | Excellent -- production ready | | 0.80-0.90 | Good -- minor improvements | | 0.60-0.80 | Needs work | | <0.60 | Poor | ## Common Issues 1. Low context precision (<0.7): Vector search returns irrelevant docs 2. Low faithfulness (<0.8): LLM adds info not in context 3. Low answer relevancy (<0.8): LLM misses the question ## RAGAS vs DeepEval | Feature | RAGAS | DeepEval | |---------|-------|----------| | Stars | 14,855 | 16,885 | | Focus | RAG pipeline | General LLM | | Metrics | 6 core | 14+ | | CI/CD | Manual | Native pytest | ## Bottom Line Start with RAGAS for essential metrics. Add DeepEval for advanced features. All data from GitHub API on 2026-07-16.