RAG Evaluation

What it is

RAG evaluation is measuring, with numbers, whether the system retrieves the right chunks and whether the answer it writes is faithful to them. It replaces 'I asked it three questions and it seemed fine' with a test set you can rerun after every change.

There are two separate things to measure. Retrieval quality: did the chunk that contains the answer make it into the top k? Generation quality: given those chunks, is the answer correct, grounded in them, and complete? A failure in either one looks the same to the user, so you measure them separately to know what to fix.

The practical version, which I use, is a list of real questions, each with the chunk or keywords that should appear, and a script that reports recall at k and keyword hit rate. The fancier version adds an LLM judge that scores faithfulness and relevance, which is what the RAGAS framework automates.

Why it's needed

Every change to a RAG system, a new chunk size, a different k, a reworded prompt, a model upgrade, can make some answers better and others worse. Without a test set you only find out when a user complains. With one, you know in sixty seconds.

Evaluation is also the thing that separates an AI engineer from someone who calls an API. Anyone can wire up retrieval and a prompt. Showing a table of recall at 3 versus recall at 5, and explaining why you picked 5, is the evidence that you engineered the system rather than assembled it.

How it works

• Build a test set: 20 to 50 real or realistic questions. For each, record the expected chunk (or a unique substring of it) and a few keywords the answer must contain. Include questions the system should refuse.
• Retrieval metrics: recall at k (is the expected chunk in the top k?), precision at k (what fraction of the top k is relevant?) and mean reciprocal rank (how high does the right chunk sit?).
• Generation metrics: keyword hit rate as the cheap proxy, then faithfulness (every claim in the answer is supported by the retrieved context) and answer relevance (it actually addresses the question), scored by an LLM judge.
• Run the full set after every change and diff the results. A change that improves average score but breaks three questions needs a look.
• Track cost and latency per question alongside quality, so an improvement that doubles latency is visible.
• Promote production failures into the test set. Every bad answer a user reports becomes a permanent regression test.

Code example

import json, statistics
from openai import OpenAI

client = OpenAI()

TESTS = [
    {"q": "What is his tech stack?", "expect_chunk": "Django, FastAPI",
     "keywords": ["Django", "FastAPI", "AWS"]},
    {"q": "Where did he study?", "expect_chunk": "University of Central Punjab",
     "keywords": ["University of Central Punjab"]},
]

def judge_faithful(question, context, answer) -> int:
    """LLM-as-judge: 1 if every claim is supported by the context."""
    r = client.chat.completions.create(model="gpt-4o-mini", messages=[{
        "role": "user", "content":
        f"Context:\n{context}\n\nQuestion: {question}\nAnswer: {answer}\n\n"
        "Is every factual claim in the answer supported by the context? "
        "Reply with only 1 (yes) or 0 (no)."}])
    return int(r.choices[0].message.content.strip()[0])

def evaluate(retrieve, answer, k=5):
    recall, hits, faith = [], [], []
    for t in TESTS:
        chunks = retrieve(t["q"], k)
        recall.append(any(t["expect_chunk"] in c for c in chunks))
        a = answer(t["q"], chunks)
        hits.append(sum(kw.lower() in a.lower() for kw in t["keywords"]) / len(t["keywords"]))
        faith.append(judge_faithful(t["q"], "\n".join(chunks), a))
    print(json.dumps({"recall@k": statistics.mean(recall),
                      "keyword_hit": statistics.mean(hits),
                      "faithfulness": statistics.mean(faith)}, indent=2))

Where I used this

Use cases

• Regression testing after changing prompts, chunking, k or the model
• Choosing between embedding models or retrieval strategies with evidence
• Reporting quality to a client in numbers rather than anecdotes
• Catching drift when the knowledge base is updated
• Comparing cost and latency against quality for a model upgrade

Common mistakes

• A test set written by the developer in one sitting. It contains the questions you already know work. Pull real questions from logs.
• Measuring only the final answer, so you cannot tell whether a miss came from retrieval or generation.
• Using the same model as judge and generator without checking the judge. Spot-check a sample of judge scores by hand first.
• Chasing the average. A system that scores 90 percent while failing the same five questions every time has a specific bug worth more than the average.
• Never rerunning. An evaluation script that runs once before launch is a demo, not a safety net. Wire it into your workflow.

Interview questions