RAGAS 2026 Tutorial: Set Up LLM-Powered RAG Evaluation in 30 Minutes

RAGAS is the standard open-source framework for RAG evaluation, but the docs assume you know its concepts. Here is a working setup: metrics, code, and how to interpret the scores.

💡 What You Will Learn

RAGAS is the standard open-source framework for RAG evaluation, but the docs assume you know its concepts. Here is a working setup: metrics, code, and how to interpret the scores.

📜 Table of Contents

• What RAGAS Is
• What You Need
• The Working Setup
• How to Read the Scores
• The Production Integration
• The Practical Warnings
• The 30-Minute Plan

What RAGAS Is

RAGAS (Retrieval Augmented Generation Assessment) is the open-source framework that popularized LLM-judge-based RAG evaluation. It computes the metrics from the RAG metrics guide - faithfulness, answer relevancy, context precision/recall - using a judge LLM instead of human labels.

What You Need

• A test set: 50-200 questions with the answer context (or full docs)
• A judge LLM: any strong model (OpenAI, Claude, DeepSeek, or a local model via Ollama)
• Your RAG pipeline: to generate answers on the test set

The Working Setup

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision
from datasets import Dataset

# 1. Run your RAG on the test questions and collect:
#    question, answer, contexts (retrieved chunks), ground_truth (optional)
data = {
    "question": ["What is the refund policy?"],
    "answer": ["Refunds are available within 30 days."],
    "contexts": [["Refunds within 30 days of purchase."]],
    "ground_truth": ["30-day refund policy."],
}
dataset = Dataset.from_dict(data)

# 2. Evaluate
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy, context_precision])
print(result)

That is the whole loop. RAGAS calls the judge LLM for each metric and returns 0-1 scores per question plus an aggregate.

How to Read the Scores

• Faithfulness 0.9 - the answer sticks to retrieved context; hallucination risk low
• Answer relevancy 0.6 - the answer partially misses the question; check query understanding
• Context precision 0.4 - lots of noise in retrieval; tighten chunking or add reranking (see the chunking guide)

Look at per-question scores, not just the average - one broken retrieval can hide in a good mean.

The Production Integration

1. Run RAGAS on a golden set in CI on every RAG change - catches regressions before deploy.
2. Score sampled production traffic with the same metrics for drift (see the evaluation platforms guide for the full architecture).
3. Keep the judge LLM fixed across runs, or score changes become judge changes.

The Practical Warnings

• Judge cost: 3 metrics x N questions x tokens - keep the test set tight (100 questions is plenty).
• Judge bias: use a different model family for judging than the one generating answers.
• Ground truth is optional for faithfulness/relevancy but required for answer correctness metrics - start without it, add it for the metrics you care about.

The 30-Minute Plan

Minutes 1-10: collect 50 real questions. Minutes 10-20: run your RAG, dump question/answer/contexts to a dataset. Minutes 20-30: evaluate with the code above and read the per-question scores. You now have a quantitative RAG quality baseline - the thing most teams never build.