RAGAS 2026 Tutorial: Set Up LLM-Powered RAG Evaluation in 30 Minutes
RAGAS 2026 Tutorial: Set Up LLM-Powered RAG Evaluation in 30 Minutes。After changing chunking, embeddings and reranking, most teams eyeball a few answers and ship. RAGAS turns that
📜 Table of Contents
RAGAS Tutorial 2026: A Working RAG Evaluation in 30 Minutes
After changing chunking, embeddings and reranking, most teams eyeball a few answers and ship. RAGAS turns that feeling into numbers by using a judge LLM to score each answer.
What it measures
| Metric | Question | Needs ground truth |
|---|---|---|
| Faithfulness | Is every claim supported by retrieved context? | No |
| Answer relevancy | Does it address the question? | No |
| Context precision | How much retrieved content is useful | Yes |
| Context recall | How much of the needed content was found | Yes |
| Answer correctness | Agreement with the reference answer | Yes |
Faithfulness is the most valuable: it decomposes the answer into claims and checks each against context, which makes hallucinations visible.
Three inputs
A test set of 50-200 real user questions, a strong judge LLM (cloud or local), and a batch-runnable RAG pipeline that returns question, answer and contexts. Ground truth is only required for some metrics, so a hallucination check works without labels.
Minimal code
Collect question, answer, contexts, ground_truth per sample into JSON, load with datasets.Dataset.from_list, then call evaluate(dataset=..., metrics=[faithfulness, answer_relevancy, context_precision, context_recall]) and export result.to_pandas() to CSV. Judge and embedding setup differs across versions, including wrapper classes, so check the official docs for your installed version. Ollama can serve as judge with a base_url and model name.
Reading scores
| Symptom | Likely cause | Fix first |
|---|---|---|
| Low faithfulness | Fabrication or forced answers | Instruct refusal, block empty retrieval |
| Low relevancy | Misread question, polluted history | Query rewriting, history compression |
| Low context precision | Noisy retrieval | Tighter chunks, thresholds, reranking |
| Low context recall | Missing content | Better embeddings, hybrid search |
| Everything low | Bad test set | Re-examine questions and labels |
Inspect per-question scores, not averages. Ninety 0.95s and ten 0.2s look fine on average and terrible to users.
Make it routine
Re-run the golden set after every retrieval or prompt change, gate merges in CI on a baseline, sample production traffic daily to catch drift, and never swap the judge model without recalibrating.
Cost and traps
Judge calls bill per token, so start with 50 questions before scaling metrics. Do not use the same model as judge and candidate. Pass untruncated contexts. Pick a judge that is strong in your content language. Scores are comparable only within the same test set.
FAQ
Evaluate without ground truth? Yes, faithfulness and relevancy work unlabeled.
Why is faithfulness always 1.0? Short, extractive answers score high; check relevancy and recall too.
Is a local judge reliable? Small models fluctuate between runs; use a larger one and fix randomness.
Alternatives? DeepEval for test-style assertions, Arize Phoenix for tracing and observability.
Related
Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.