RAGAS 2026 Tutorial: Set Up LLM-Powered RAG Evaluation in 30 Minutes
RAGAS is the standard open-source framework for RAG evaluation, but the docs assume you know its concepts. Here is a working setup: metrics, code, and how to interpret the scores.
💡 What You Will Learn
RAGAS is the standard open-source framework for RAG evaluation, but the docs assume you know its concepts. Here is a working setup: metrics, code, and how to interpret the scores.
📜 Table of Contents
What RAGAS Is
RAGAS (Retrieval Augmented Generation Assessment) is the open-source framework that popularized LLM-judge-based RAG evaluation. It computes the metrics from the RAG metrics guide - faithfulness, answer relevancy, context precision/recall - using a judge LLM instead of human labels.
What You Need
- A test set: 50-200 questions with the answer context (or full docs)
- A judge LLM: any strong model (OpenAI, Claude, DeepSeek, or a local model via Ollama)
- Your RAG pipeline: to generate answers on the test set
The Working Setup
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset
# 1. Run your RAG on the test questions and collect:
# question, answer, contexts (retrieved chunks), ground_truth (optional)
data = {
"question": ["What is the refund policy?"],
"answer": ["Refunds are available within 30 days."],
"contexts": [["Refunds within 30 days of purchase."]],
"ground_truth": ["30-day refund policy."],
}
dataset = Dataset.from_dict(data)
# 2. Evaluate
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
print(result)
That is the whole loop. RAGAS calls the judge LLM for each metric and returns 0-1 scores per question plus an aggregate.
How to Read the Scores
- Faithfulness 0.9 - the answer sticks to retrieved context; hallucination risk low
- Answer relevancy 0.6 - the answer partially misses the question; check query understanding
- Context precision 0.4 - lots of noise in retrieval; tighten chunking or add reranking (see the chunking guide)
Look at per-question scores, not just the average - one broken retrieval can hide in a good mean.
The Production Integration
- Run RAGAS on a golden set in CI on every RAG change - catches regressions before deploy.
- Score sampled production traffic with the same metrics for drift (see the evaluation platforms guide for the full architecture).
- Keep the judge LLM fixed across runs, or score changes become judge changes.
The Practical Warnings
- Judge cost: 3 metrics x N questions x tokens - keep the test set tight (100 questions is plenty).
- Judge bias: use a different model family for judging than the one generating answers.
- Ground truth is optional for faithfulness/relevancy but required for answer correctness metrics - start without it, add it for the metrics you care about.
The 30-Minute Plan
Minutes 1-10: collect 50 real questions. Minutes 10-20: run your RAG, dump question/answer/contexts to a dataset. Minutes 20-30: evaluate with the code above and read the per-question scores. You now have a quantitative RAG quality baseline - the thing most teams never build.
