RAG evaluation: gold question sets, retrieval accuracy and faithfulness

You edit the prompt, change how chunks are split, swap the embedding model, and the chatbot "seems better". But better by how much, on which questions, and which questions are quietly getting worse? Without a proper RAG evaluation, every change is a gut feeling. This article builds a compact RAG evaluation workflow that runs on a laptop: a golden question set, retrieval metrics you implement yourself, faithfulness checks with a rule and with an LLM judge, and then shows exactly where these measurements get it wrong.

It reuses the system from Building RAG in Python: chunking, retrieval and cited answers and follows Advanced RAG: hybrid search, reranking and query rewriting: that article added several techniques, and this one answers the question "which technique is actually worth it" with RAG evaluation numbers. Environment: Apple M4, 16 GB RAM, Python 3.12, the openai SDK pointed at Ollama through its OpenAI-compatible endpoint http://localhost:11434/v1, models qwen2.5:1.5b and nomic-embed-text; the code lives in code/s7-39.

Why RAG evaluation must be split into two layers

A RAG system has two parts that can fail independently: retrieval (finding the right passage) and generation (using that passage to answer correctly without adding anything). If you only grade the final answer, a wrong answer could come from a retrieval miss, from the model misreading the right passage, or from both, and you will fix the wrong thing. Literature on RAG evaluation splits the problem this way. The paper Ragas: Automated Evaluation of Retrieval Augmented Generation (Es et al., 2023) names the dimensions: whether retrieval finds relevant and focused passages, whether the model uses them faithfully, and the quality of the generated answer. The ARES paper (Saad-Falcon et al., NAACL 2024) uses three similar dimensions: context relevance, answer faithfulness and answer relevance.

One caveat for RAG evaluation: faithful (sticks to the context) is not the same as correct (matches reality). An answer can follow an outdated passage closely and still be wrong, so RAG evaluation needs both kinds of metrics.

Building a golden question set for RAG evaluation

The golden set is the most valuable asset in RAG evaluation, more than any framework. Each sample has a question, a type (answerable or out of corpus), a gold section (the file and heading that contain the answer) and the facts that must appear in the answer. The test corpus for this RAG evaluation is the 11 policy documents of an imaginary home-appliance store from the previous RAG article. Principles for building it:

• Take real questions from tickets and chats; questions generated by a model from the documents often share vocabulary with them and make retrieval look better than it is.
• Include out-of-corpus questions (roughly 10 to 20%) to test the ability to refuse.
• Label with gold sections instead of chunk text, so you can change the chunking without relabeling.
• Freeze and version it before optimizing, so you do not edit the test set to fit the system.

The set in this article has the 25 questions from the previous RAG article plus 7 new ones (make_golden.py; the 25 old questions were copied to questions.json next to the script so this article's code folder runs on its own):

32 questions: 27 answerable from the corpus, 5 out of corpus

Retrieval metrics for RAG evaluation: implement them to understand every number

Retrieval metrics in RAG evaluation are all computed from a list of binary flags: is the chunk at rank i relevant or not. They are short enough that you should write them yourself rather than depend on a library, and writing them shows you exactly what you are measuring. They are also the familiar measures from Model evaluation for classifiers: precision, recall, ROC and imbalanced data, applied to a ranking problem.

def hit_at_k(rels, k):          # is there a relevant chunk in the top-k?
    return float(any(rels[:k]))

def mrr(rels):                  # reciprocal rank of the first relevant chunk
    return next((1 / i for i, r in enumerate(rels, 1) if r), 0.0)
# precision@k = sum(rels[:k]) / k; nDCG@k = DCG / ideal DCG (full set in metrics.py)

Check by hand with the case where relevant chunks are at rank 2 and rank 4 (rels = [0, 1, 0, 1, 0]):

hit@1 0.0 hit@3 1.0
precision@3 0.333 MRR 0.5
nDCG@5 0.651

Reading the result: hit@3 is 1 because a correct chunk is in the first 3 ranks, and MRR is 1/2 because the first correct chunk is at rank 2. hit@k tells you whether the generator even gets a chance to see the right passage, so it is the main RAG metric when k is small; MRR and nDCG matter when order affects the result. If each question has only one gold section, precision@3 cannot exceed about 1/3 even for perfect retrieval, so do not use it as the main metric for question answering.

RAG evaluation of retrieval: three configurations and a baseline

We compare three chunking configurations on the 27 answerable questions: fixed 400 characters, splitting by H2 section (at most 900 characters), and by section with a threshold of 350. A section chunk counts as relevant when it has the right file and gold section; a fixed chunk counts when it is in the right file and contains all the facts, so the difference between the two kinds is only a trend. We add a random baseline (random ranking, repeated 200 times) to see whether the numbers mean anything.

27 answerable questions
configuration          chunks  hit@1   hit@3     P@3     R@3     MRR  nDCG@5
fixed 400 chars           48   0.519   0.741   0.259   0.741   0.630   0.647
by section, ≤900          51   0.704   0.852   0.284   0.852   0.790   0.824
by section, ≤350          52   0.741   0.852   0.296   0.852   0.809   0.835
random (avg of 200 runs)  51   0.017   0.063   0.021   0.063   0.045   0.059

Three observations. In this RAG evaluation the random baseline gives hit@3 of about 0.06 (roughly 3/51), so 0.85 is a real signal. Precision@3 sits around 0.28 to 0.30, as predicted. The threshold of 350 creates only one extra chunk, so hit@1 rising from 0.704 to 0.741 is equivalent to one more correct question (20/27 instead of 19/27): that is noise, not an improvement.

RAG evaluation with a small set: bootstrap confidence intervals

With 27 questions, every question that flips state moves hit@k by almost 4 percentage points. The bootstrap resamples the 27 hit/miss flags with replacement, recomputes the rate 2000 times, and takes the 2.5% and 97.5% percentiles (ci_and_k.py).

hit@3 = 0.852 on 27 questions; 95% confidence interval (bootstrap) = [0.70, 0.96]

The interval [0.70, 0.96] is very wide: a difference of a few points between two configurations on this set is indistinguishable from noise. To draw firm conclusions in RAG evaluation you need a few hundred questions, or a paired comparison on the same questions instead of comparing two separate rates.

End-to-end RAG evaluation: retrieval, facts, citations, refusals

Full-pipeline RAG evaluation: run the entire RAG pipeline from the previous article (top-3, section chunks, JSON output with answer and sources) on all 32 questions, then grade each layer separately (eval_generation.py) to put the blame in the right place.

answerable (27 questions)
  retrieved the right section in top-3 : 23/27
  answer contains all required facts   : 21/27
    among correctly retrieved          : 21/23
  wrongly refused                      : 1/27
  citation hits the right section      : 21/27
  has a number not in the context      : 1/27
out of corpus (5 questions)
  correctly refused                    : 5/5
  invented numbers                     : 0/5

The four retrieval misses (Q15, Q16, Q19, Q22) all got wrong answers: when the right passage is not in the context, a small model cannot know the answer on its own. The rest is the generation layer's job: 21 of the 23 correctly retrieved questions have all the facts. But those two "wrong" ones need a closer look, because the RAG evaluation metric itself has errors too.

When the RAG evaluation metric measures wrongly: inspect each case

The log lists the questions that were retrieved correctly but graded as missing facts, and the questions flagged with a strange number:

Wrong-fact answers despite correct retrieval:
 - Q06 Is a machine with water damage to the motor covered by warranty?
   -> No
 - Q11 What is the maximum COD order value?
   -> COD orders of 3,000,000 VND or more need confirmation by message or call before the warehouse ships.

Questions with an unfamiliar number:
 - Q26 ['4'] :: A COD order of 4 million VND needs confirmation by message or call before the warehouse ships.

• Q06 is a false positive of the metric. The answer "No" is perfectly right (the document says water damage to the motor section means the warranty claim is denied), but the fact I set was the string "denied", so it was graded as a miss. Fact string matching is sensitive to phrasing; you must review the failed cases by hand before trusting the number.
• Q11 is a real generation error. The right passage is in the context, but the 1.5-billion-parameter model picked the wrong sentence, "from 3,000,000 VND needs confirmation", instead of "up to 5,000,000 VND". The answer is faithful (3,000,000 appears in the context) but does not answer the question. The number check cannot catch this, and it is why faithfulness is not enough.
• Q26 is a false alarm from the number-check rule. The number "4" comes from the question itself ("4 million"), and since it is not in the documents it was flagged, even though the answer is correct. The rule needs to ignore numbers repeated from the question, or normalize "4 million" to 4,000,000.

After the manual review, the true RAG evaluation result is 22/27 correct instead of 21/27: four wrong because of retrieval, one wrong because of generation (Q11). The lesson: the automatic number is only a starting point, and after every system change you should manually review all the cases that changed state, not just look at the total.

Citations also need their own measurement: 21/27 hit the right section, of which 4 missed because of retrieval, while Q03 and Q26 answered correctly but the model wrote sources: [2] when the right passage was [1]. A reader who clicks a source and finds nothing matching is a dangerous failure, so measure citation accuracy as a separate metric.

RAG evaluation of generation: faithfulness with a rule and an LLM judge

In Ragas-style RAG evaluation, faithfulness is computed by splitting the answer into claims, checking whether each claim can be inferred from the context, then dividing the number of supported claims by the total number of claims (Ragas documentation on Faithfulness). An LLM performs the checking step. We implement exactly this idea with a local qwen2.5:1.5b judge (judge.py), and set it beside a much cheaper rule (grounding.py): every number in the answer must appear in the context.

The most important question about any judge: does it catch real errors? You need data with known labels: take 20 answers already verified by hand as correct, then create an "injected error" version by doubling the first number and adding 3 ("7 days" becomes "17 days"). That makes 39 samples, given to the judge and to the number-check rule:

calibration set: 39 answers (20 faithful, 19 with altered numbers)
judge qwen2.5:1.5b : correctly accepted faithful answers 20/20, caught altered answers 14/19
overall accuracy   : 0.87
number-check rule  : accuracy 0.97

Examples where the judge was wrong:
 - Q01 label=altered judge=YES :: Customers can exchange or return a product within 17 days of receiving it if it still has its tag, box and accessories and has not been used.
 - Q02 label=altered judge=YES :: 60003 VND per return order.
 - Q13 label=altered judge=YES :: Orders from 6000003 VND
 - Q17 label=altered judge=YES :: 5 parts rice and 1.8 parts water

Two takeaways. The 1.5-billion-parameter judge never misses a faithful answer, but it lets 5 of the 19 altered answers slip through (about 26%): it tends to nod along when a claim looks close to the context. In this RAG evaluation the number-check rule reaches 0.97, missing only Q17 because the altered number ("5") happens to appear elsewhere in the context. But the rule cannot detect "right number, wrong subject" errors like Q11, and injected errors are the easiest kind, so 0.97 is an upper bound.

For LLM-based RAG evaluation, a stronger judge has its own biases. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., NeurIPS 2023) documents position bias, verbosity bias and self-enhancement bias toward answers generated by the same model, and also reports that GPT-4 agrees with humans over 80% of the time. The consequence: calibrate the judge on your own labeled set, use a different model from the generator, and keep a small sample for humans to check. ARES uses a small number of human labels to correct judge error (prediction-powered inference).

RAG evaluation with out-of-corpus questions: the ability to refuse

On the 5 out-of-corpus questions (an iPhone, the price of a product code that is not in the documents, hiring, shipping to Laos, an air purifier), the system refused correctly 5/5 and invented no numbers. But this set is very small, and the cosine scores of the out-of-corpus questions (0.762 to 0.812) are mixed in with the range for in-corpus questions (0.730 to 0.839) as the previous article warned, so the refusals come from the prompt and not from a score threshold. Also measure the opposite error: wrongful refusal (1/27 answerable questions), because a system that refuses too much looks safe but is useless.

Note that this metric matches a fixed refusal string in the code, so 1/27 is a lower bound: for Q15 and Q16 (retrieval misses) the model answered "No information ..." in different wording, which is really a wrongful refusal but is not counted. Counting both, wrongful refusals are at least 3/27.

A RAG evaluation workflow in a real project

1. Freeze the RAG evaluation golden set (30 to 50 questions to start, a few hundred once the system is stable) and keep it in Git next to the code.
2. Run the RAG evaluation automatically on every change to chunking, embeddings, prompt or model, then compare the table with the previous run.
3. Report by layer: hit@k and MRR for retrieval; faithfulness, fact coverage and citations for generation; correct and wrongful refusals for out-of-corpus questions.
4. Manually review the cases that changed state between two RAG evaluation runs; fix the metric if it is wrong before fixing the system.
5. Add new questions from real user failures, but keep a separate held-out set that you never optimized against.

Limits of the RAG evaluation workflow in this article

• 27 answerable questions is too small (the hit@3 confidence interval is 26 points wide), and each question has only one gold section, so it cannot yet measure questions that need several passages combined.
• The questions were written by the person who wrote the corpus, so they share vocabulary with the documents; questions from real customers are vaguer, and real numbers will be lower.
• A 1.5-billion-parameter judge only illustrates the method: catching 14/19 injected errors is not enough for a deployment gate.
• The judge is the same as the generator: because of resource limits there is only one chat model (qwen2.5:1.5b), so judge and generator are one, against the advice above; self-enhancement bias may skew the 20/20 and 14/19 figures.
• The refusal metric matches a fixed string, so it misses refusals worded differently (Q15, Q16), as noted in the out-of-corpus section.
• Every RAG evaluation command was actually run with local models; there are no numbers from commercial APIs.

RAG evaluation checklist

• Have a golden set labeled with gold sections and facts, including out-of-corpus questions.
• Split RAG evaluation by layer: retrieval (hit@k, MRR) separately, generation (fact coverage, faithfulness, citations) separately.
• Compare against a random baseline and include confidence intervals before drawing conclusions.
• Calibrate every judge with a sample of injected errors; manually review cases that change state.
• Report correct refusals and wrongful refusals side by side.

Source code: make_golden.py, metrics.py, eval_retrieval.py, eval_generation.py, grounding.py, judge.py, ci_and_k.py, plus golden.json and the *_output.txt files. The numbers are real output on an Apple M4, 16 GB, temperature=0, seed=7; a different Ollama version may shift a few answers.

Next article

Structured data extraction from HTML with LLMs and Pydantic. The next article moves from question answering to turning messy text into typed records, validated with Pydantic, and reuses today's RAG evaluation mindset: a gold sample set, measuring each field, and measuring the errors of the metric itself.