Ragas 0.4 Tutorial: An Evaluation Harness That Splits Retrieval Failures from Generation Failures

The assistant answers every question in fluent paragraphs that nobody can trace back to a document. Retrieval returns the wrong chunk, the model ignores the right one, and both look identical in the logs, so the team argues about prompts while the real bug sits in chunking.

💡 What You Will Learn

The assistant answers every question in fluent paragraphs that nobody can trace back to a document. Retrieval returns the wrong chunk, the model ignores the right one, and both look identical in the l

📜 Table of Contents

• What is in the box right now
• Four numbers, four different bugs
• The signatures, exactly
• Build the eval set from your own logs
• Run it as a batch, not one row at a time
• Turn scores into a gate
• What a faithfulness run actually costs
• Four traps that make the numbers meaningless
• Where the batch API still fits
• FAQ

A RAG system that answers confidently and wrongly is easy to build. The hard part is knowing which of the two halves broke. Retrieval that returns the wrong chunk and a model that ignores the right chunk produce the same symptom from the outside: a fluent paragraph that no one can trace back to a document. Ragas exists to split that symptom into numbers, and version 0.4 changed how you call it.

This walkthrough builds a small evaluation harness against your own documents, using the pieces that ship in ragas 0.4.3 as of 2026-10-24. Every import path and metric signature below was read out of the published wheel, not from memory.

What is in the box right now

• ragas 0.4.3 on PyPI, Python 3.9 and newer. The repository now lives under vibrantlabsai/ragas; the old explodinggradients/ragas URL still resolves, which is why half the tutorials you find link to the old name.
• Two APIs in the same package. The modern one is ragas.metrics.collections, where each metric is a class you construct with an LLM and call with await metric.ascore(...). The batch one is still exported at the top level: from ragas import evaluate, aevaluate, EvaluationDataset, SingleTurnSample. Both work in 0.4.3.
• A CLI. ragas quickstart rag_eval scaffolds a complete evaluation project if you would rather start from their layout than from this article.
• Caching. DiskCacheBackend and the cacher decorator are exported from ragas. Without them, every re-run pays full judge-model cost again.

Four numbers, four different bugs

Pick metrics because each one can only fail for one reason. That constraint is what makes a dashboard useful.

• Context recall asks whether the retrieved chunks contained everything the reference answer needed. A low score is a retrieval bug: chunking, embeddings, or the query itself. No prompt change will fix it.
• Context precision asks whether the useful chunks were ranked near the top. Low precision with high recall means the answer is in there, buried under three pages of noise, which is why long-context models answer worse than their context window suggests.
• Faithfulness breaks the answer into atomic claims and checks each one against the retrieved context. Low faithfulness with high recall is the most expensive failure mode: the information was present and the model invented something instead.
• Answer relevancy asks whether the answer addressed the question. This one catches the assistant that answers a neighbouring question very well.

Read them as a matrix. Recall low, faithfulness high: fix retrieval. Recall high, faithfulness low: fix generation, usually by tightening the instruction to quote and by lowering temperature. Both high, relevancy low: the prompt is answering the wrong thing.

The signatures, exactly

These are the keyword arguments the 0.4.3 wheel requires. Guessing them costs an afternoon of TypeError, so they are worth printing once:

from ragas.metrics.collections import (
    Faithfulness, AnswerRelevancy, ContextPrecision, ContextRecall,
)

Faithfulness(llm=judge).ascore(
    user_input="...question...", response="...answer...",
    retrieved_contexts=["chunk 1", "chunk 2"])
AnswerRelevancy(llm=judge).ascore(user_input="...", response="...")
ContextRecall(llm=judge).ascore(
    user_input="...", retrieved_contexts=[...], reference="...gold answer...")
ContextPrecision(llm=judge).ascore(
    user_input="...", reference="...gold answer...", retrieved_contexts=[...])

Note the asymmetry: recall takes the reference answer, precision takes the reference too but orders its arguments differently. Faithfulness needs the retrieved contexts and nothing else. The judge is created with llm_factory:

from openai import AsyncOpenAI
from ragas.llms import llm_factory

client = AsyncOpenAI()
judge = llm_factory("gpt-4o-mini", client=client)

llm_factory returns an instructor-backed model that produces structured output, which is what the collections metrics insist on. Legacy LangChain-wrapped LLMs are rejected here with an explicit error rather than silently scoring garbage, and the wrapper classes are marked deprecated in favour of llm_factory.

Build the eval set from your own logs

Synthetic questions generated from your documents are systematically easier than the questions real users type. Ragas ships test-set generation and it is fine for a first pass, but a harness built on production traffic will find bugs that generation never invents.

The rule that saves the most time later: capture the retrieved contexts at query time and store them next to the question. If you re-run retrieval when you evaluate, you are measuring a different system than the one that served the user, and quiet index changes will look like model regressions.

# eval_set.jsonl, one row per answered question
{"question": "...", "answer": "...", "contexts": ["...", "..."], "reference": "..."}

Twenty to fifty rows is enough to see a direction, provided the rows are real. Tag each row with the document family it came from so a bad score can be traced to one section of the corpus rather than to "the RAG".

Run it as a batch, not one row at a time

import asyncio, json
from ragas.metrics.collections import Faithfulness, ContextRecall

async def score_row(row, sem, faith, recall):
    async with sem:
        f = await faith.ascore(user_input=row["question"], response=row["answer"],
                               retrieved_contexts=row["contexts"])
        r = await recall.ascore(user_input=row["question"],
                                retrieved_contexts=row["contexts"],
                                reference=row["reference"])
        return {**row, "faithfulness": f.value, "context_recall": r.value}

async def main():
    rows = [json.loads(line) for line in open("eval_set.jsonl")]
    sem = asyncio.Semaphore(4)
    faith, recall = Faithfulness(llm=judge), ContextRecall(llm=judge)
    results = await asyncio.gather(*(score_row(r, sem, faith, recall) for r in rows))
    with open("scores.jsonl", "w") as fh:
        for r in results:
            fh.write(json.dumps(r) + "\n")

asyncio.run(main())

The semaphore matters more than it looks. Four concurrent judge calls is polite to a hosted API and keeps failures distinguishable from rate limits. Set it to fifty and half your rows come back as errors that look like model failures.

Turn scores into a gate

A dashboard nobody reads changes nothing. Two thresholds, checked in CI on every prompt or index change, are enough to start:

• Context recall below 0.70 on the whole set: retrieval ticket. Do not touch the prompt.
• Faithfulness below 0.85 while recall is above 0.80: generation ticket, usually instruction and temperature.

Pick the numbers from your own baseline rather than from this article. Run the harness once, record the scores, then treat a five-point drop as a regression. Absolute score levels are not comparable across corpora, and a team that copies someone else thresholds spends a month chasing a gap that means nothing.

What a faithfulness run actually costs

Faithfulness is not one call per answer. It generates atomic statements from the answer, then checks each statement against the context with an NLI call. A twelve-claim answer is roughly thirteen judge calls. Fifty answers at that length is about 650 calls for one metric, which is why teams that retry the whole set on every commit quietly stop running evals. Cache the judge results, or restrict the full set to nightly runs and a ten-row smoke subset to pre-commit.

Four traps that make the numbers meaningless

• A judge weaker than the system. If the answerer runs a frontier model and the judge runs a small one, low faithfulness often means the judge failed to follow the claim, not that the answer hallucinated. Use a judge at least as capable, or hand-check a sample to calibrate.
• Passing cleaned-up contexts. Feed the retriever output verbatim. Passing a tidied paragraph from the source document inflates recall and hides the chunking bug you were looking for.
• Judging with temperature above zero. The same row scores differently on two runs, and the team concludes the metric is noise. Pin it.
• Treating scores as truth. These are LLM judgments with error bars. Label twenty rows yourself, compare with the metric, and write down the agreement rate. When a metric disagrees with your labels more than one time in five, fix the eval before trusting the dashboard.

For rubrics that are not in the box, DiscreteMetric takes a prompt and an allowed_values list and returns a label with a reason, which covers things like "does this answer cite the right policy number". Keep those custom metrics small and specific; a metric that tries to score overall quality ends up measuring style.

Where the batch API still fits

The dataset-style path (EvaluationDataset plus evaluate or aevaluate) still exists in 0.4.3 and is the shorter route when you already have a dataframe of question, answer and context columns. Use it for reporting; use the async per-metric calls when you want one number in isolation, because a batch run that fails on row 37 tells you nothing about why.

FAQ

Do I need an OpenAI key? No, but the judge must support structured output through an instructor-compatible client. Local models work when they can reliably return the schema, and smaller local judges fail on that before they fail on judgement quality.

How many rows before the scores mean anything? Twenty real questions reveal a direction. Below ten, a single unusual answer moves the average by ten points and the harness becomes a random number generator with a CSV.

Are my prompts and data sent anywhere? Ragas sends content to whichever model you configure as judge, and collects anonymised usage analytics by default. Set RAGAS_DO_NOT_TRACK=true if your policy requires it.

Should evaluation live in CI? The ten-row smoke subset yes, the full set no. Full runs go nightly, with the score history committed so regressions have a date.

Once the numbers separate retrieval from generation, the next common failure is orchestration rather than quality; the LangGraph supervisor walkthrough covers routing bugs, and the PDF to Markdown harness covers the ingestion step that quietly ruins recall before a single query runs.