Evaluating RAG

The big idea

A RAG answer can be wrong for two very different reasons: retrieval never found the right passage, or the model had the right passage and still answered badly. Each needs a different fix, so measure them separately. If you only score final answers, you cannot tell whether to tune chunking and search or the prompt and model.

Retrieval is measured against a golden setA curated set of real questions with known correct answers or known source passages, used as a fixed benchmark for a system.: real questions, each labelled with the chunks that contain its answer. Then you check whether those chunks come back near the top. Generation is measured on two properties: faithfulnessWhether every claim in an answer is supported by the retrieved context, rather than invented or taken from the model's memory., whether every claim is supported by the retrieved text, and relevance, whether the answer actually addresses the question.

Think of a student answering an open-book exam. If they get a question wrong, you want to know: did they look in the wrong chapter, or did they read the right page and misunderstand it? Grading only the final answer tells you neither.

How it works

Building a golden set

Start from real questions: search logs, support tickets, questions people asked in a pilot. Synthetic questions generated from your documents help fill gaps, but they tend to reuse the documents' wording, which makes retrieval look better than it is. For each question, record the chunk or chunks that contain the answer (or the documents and sections, which survive re-chunking), and ideally a reference answer.

Fifty to a few hundred questions is enough to start. Cover the categories that matter: common questions, rare ones, questions that need two sources, questions the corpus cannot answer (where the right behaviour is to say so), and the failures users reported. Keep adding real failures; the set grows with the product ( Building an eval set ).

Retrieval metrics

• Recall@kThe share of a question's relevant chunks that appear in the top k results, averaged over questions.: of the relevant chunks, what share appear in the top k? This is the most important retrieval metric for RAG, because the model can only use what reaches it. Measure it at the k you actually send to the model, and at the larger k you rerank.
• Hit rate@k: did at least one relevant chunk appear in the top k?
• Precision@k: what share of the top k is relevant? Low precision means the model wades through noise.
• MRRMean reciprocal rank: the average over questions of 1 divided by the rank of the first relevant result, so 1 means always first and 0.5 means typically second. (mean reciprocal rank): the average of 1 / (rank of the first relevant result). It rewards putting the answer first.

Generation metrics

• Faithfulness: split the answer into claims and check each against the retrieved context. An unsupported claim is a hallucination even if it happens to be true.
• Answer relevance: does it answer what was asked, completely and without padding?
• Correctness: when you have reference answers, does the answer match them?
• Refusal behaviour: for questions the corpus cannot answer, does the system say so instead of guessing?

Who grades

People are the gold standard but slow. An LLM judgeUsing a language model, prompted with a rubric, to grade outputs such as answers, for example for faithfulness or relevance. prompted with a clear rubric ("for each claim, quote the supporting sentence or mark it unsupported") scales to every change. Validate the judge on a sample graded by people, track how often they agree, and re-check after changing the judge's prompt or model ( LLM-as-judge ).

Try it

The rankings are illustrative.

Try this:

1. With hybrid search at k = 3, recall@3 is 0.56 and MRR 0.52. Switch to the reranker: recall@3 rises to 0.88 and MRR to 0.81. Which questions moved?
2. Set k to 10 and compare the systems. Recall is 0.88 for both: the reranker only reorders the same candidates, so it improves the top, not the total. "invoice in euros" has no relevant chunk in the top 10 for either; that question needs an ingestion or first-stage fix.
3. Slide k from 1 to 10 and watch precision fall while recall rises. If you send 10 chunks to the model, most of what it reads is irrelevant; if you send 1, you miss answers. The k you choose is a trade-off you can now measure.

Going deeper

Both halves in code

The judge here is a crude word-match stand-in; in practice an LLM judge or a person does this step.

# Evaluate the two halves of RAG separately: retrieval against a golden set, then the answer's
# faithfulness to the retrieved text. `supported` stands in for an LLM judge or a person.
import re

golden = {  # question -> ids of the chunks that contain the answer
    "refund after 14 days": {"c7"}, "export limit in Notes": {"c2"}, "invoice in euros": {"c9"},
    "reset 2-step sign in": {"c4", "c5"},
}
retrieved = {  # what the system returned, best first
    "refund after 14 days": ["c7", "c1", "c3"], "export limit in Notes": ["c1", "m2", "c2"],
    "invoice in euros": ["c3", "c8", "c1"], "reset 2-step sign in": ["c4", "c6", "c1"],
}

def recall_at(k):
    return sum(len(set(retrieved[q][:k]) & rel) / len(rel) for q, rel in golden.items()) / len(golden)

def mrr():
    rr = [next((1 / (i + 1) for i, c in enumerate(retrieved[q]) if c in rel), 0) for q, rel in golden.items()]
    return sum(rr) / len(rr)

print(f"recall@1 {recall_at(1):.2f}  recall@3 {recall_at(3):.2f}  MRR {mrr():.2f}")

context = "After 14 days, annual plans are refunded pro rata for unused months. Refunds take 5 to 10 days."
answer = ("After 14 days you get a pro rata refund for the unused months. "
          "The refund arrives within 5 to 10 days. "
          "You can also request a full refund by contacting support within 30 days.")

def supported(claim, ctx):
    nums = re.findall(r"\d+", claim)
    words = {w for w in re.findall(r"[a-z]+", claim.lower()) if len(w) > 3}
    hits = sum(w[:5] in ctx.lower() for w in words)          # crude: shared word stems
    return all(n in ctx for n in nums) and hits / max(1, len(words)) >= 0.5

claims = [c.strip() for c in answer.split(". ") if c.strip()]
verdicts = [supported(c, context) for c in claims]
for c, v in zip(claims, verdicts):
    print("supported  " if v else "UNSUPPORTED", "|", c.rstrip("."))
print(f"faithfulness: {sum(verdicts)}/{len(claims)} claims supported by the retrieved text")

recall@1 0.38  recall@3 0.62  MRR 0.58
supported   | After 14 days you get a pro rata refund for the unused months
supported   | The refund arrives within 5 to 10 days
UNSUPPORTED | You can also request a full refund by contacting support within 30 days
faithfulness: 2/3 claims supported by the retrieved text

Retrieval misses one question entirely and ranks another's chunk third. The answer's first two claims are supported, but the third, a full refund within 30 days, appears nowhere in the context: a faithfulness failure, the classic sign of the model filling a gap from its own guesses.

Reference-free evaluation

Labelling sources for every question is work. Reference-free methods use an LLM to estimate context relevance, faithfulness and answer relevance directly from the question, the retrieved text and the answer, without labelled answers. They are useful for monitoring at scale and for quick iteration, but check them against a human-labelled sample before trusting their numbers.

Evaluate every change

Chunk size, embedding model, k, reranker, prompt: run the golden set before and after each change, and keep a table of versions and scores. Many "improvements" help one category and hurt another, which an average hides; break results down by category. Online signals, such as user thumbs and follow-up questions, complete the picture ( Online evaluation and monitoring ).

Trade-offs

Common mistakes

• Scoring only final answers, so retrieval and generation failures are indistinguishable.
• A golden set made only of synthetic questions that copy the documents' wording.
• Labelling exact chunk IDs, then re-chunking and silently invalidating the labels.
• Trusting an LLM judge without measuring its agreement with people.
• Reporting one average score and missing a category that got much worse.

In the wild

Check yourself

Recap