RAG Evaluation: Measure Retrieval and Generation Quality

RAG evaluation measures two independent systems — the retriever (does it find the right documents?) and the generator (does the LLM answer correctly from them?) — plus their integration. Teams that skip systematic RAG eval ship retrieval pipelines at 60-70% accuracy and discover it in user complaints. This guide builds a complete RAG evaluation stack: retrieval metrics, generation metrics, dataset design, RAGAS-style scoring, and a human review loop.

Why RAG Fails Without Evaluation

RAG pipelines have five failure modes that casual testing misses: retriever misses the relevant chunk (recall failure), retrieves wrong chunks (precision failure), generator ignores retrieved context (faithfulness failure), generator answers from parametric memory instead of context (hallucination), and ranking puts the right answer below the cutoff (MRR failure). Each needs its own metric and its own fix.

Retrieval Metrics

def recall_at_k(retrieved_ids, relevant_ids, k):
    """Fraction of relevant docs found in top-k retrieved."""
    top_k = set(retrieved_ids[:k])
    return len(top_k & relevant_ids) / len(relevant_ids)

def mrr(retrieved_ids, relevant_ids):
    """Reciprocal rank: how high the first relevant doc appears."""
    for i, doc_id in enumerate(retrieved_ids, 1):
        if doc_id in relevant_ids:
            return 1.0 / i
    return 0.0

def precision_at_k(retrieved_ids, relevant_ids, k):
    """Fraction of top-k retrieved docs that are relevant."""
    top_k = set(retrieved_ids[:k])
    return len(top_k & relevant_ids) / k

Build the retrieval test set from real queries: sample 200-500 production queries, manually tag the relevant document IDs. This is the single highest-value evaluation investment — it catches embedding drift, chunking regressions, and filter bugs.

Generation Metrics: Faithfulness and Answer Relevance

Retrieval quality doesn't guarantee answer quality. Generation evaluation needs:

Faithfulness

Does every claim in the answer trace to the retrieved context? LLM-as-judge checks each answer sentence against retrieved chunks:

def faithfulness_check(answer, contexts, judge):
    verdict = judge.chat(
        model="gpt-5",
        messages=[{"role": "user", "content": f"""
Score FAITHFULNESS (0-1): does each claim in the ANSWER
derive from the CONTEXTS? Penalize unsupported claims and
parametric-memory answers.
CONTEXTS: {contexts[:8000]}
ANSWER: {answer}
Return JSON: {{"score": 0.92, "unsupported": ["claim 2"]}}
"""}])
    return json.loads(verdict.choices[0].message.content)

Answer relevance

Does the answer actually address the question (regardless of faithfulness)? Irrelevant-but-true answers score high on faithfulness and low on relevance — both metrics are needed.

Building the Evaluation Dataset

1. Harvest real queries — top 500 production queries from logs; they represent your true distribution better than invented ones.
2. Tag relevance — for each query, mark the correct document IDs (or "no answer exists" cases — important for testing refusal behavior).
3. Curate adversarial cases — ambiguous queries, cross-document questions, queries that look similar but have different answers. 20% of the set should be hard cases or you'll overfit to easy wins.
4. Version everything — dataset, embedding model version, chunking config, and results together, so every regression is reproducible.

RAGAS-Style Automated Scoring

RAGAS (Retrieval-Augmented Generation Assessment) automates generation metrics by generating the test set itself from a document corpus:

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

result = evaluate(
    dataset=test_set,   # questions + ground truth contexts
    metrics=[faithfulness, answer_relevancy, context_precision],
    llm=judge_model,    # gpt-5-class judge
)
print(result)  # {'faithfulness': 0.91, 'answer_relevancy': 0.87, ...}

Run this on every change: embedding model swap, chunk size change, prompt tweak, retrieval topology change. Gate deploys on the three metrics holding above your thresholds.

The Human Review Loop

Automated metrics drift and miss nuance. Keep a human loop:

• Weekly: review 20-30 random answers, rate them 1-5, compare against automated scores to recalibrate the judge.
• Monthly: audit judge accuracy against 50 human-labeled cases (target >85% agreement).
• On metric drops: sample the failures, classify (retrieval vs generation vs dataset error), fix the root cause, not the symptom.

Common RAG Eval Pitfalls

• Testing only retrieval — a perfect retriever with a hallucinating generator still fails users.
• Judge grade inflation — LLM judges drift lenient; recalibrate monthly.
• Golden set rot — corpus changes make old relevance tags wrong; refresh quarterly.
• Ignoring "no answer" cases — refusal behavior is part of quality; test it.
• No versioning — unreproducible eval results are worthless.

Pair RAG evaluation with the general LLM test suite for the full quality picture, and see the RAG implementation guide for production architecture. Build and evaluate your RAG stack on DrAI's free tier — start here.