Building RAG in Python: chunking, retrieval and cited answers

A language model answers fluently about your store's return policy, but the "14 days" it quotes appears in none of your documents. That is why RAG (Retrieval-Augmented Generation) has become the default architecture for internal chatbots: instead of hoping the model remembers correctly, you find the relevant passages, put them in the prompt, and force the model to answer from them and cite the passage numbers it relied on. This article builds a complete RAG system in Python over 11 policy documents of an imaginary home-appliance store, and measures every decision with real numbers instead of gut feeling.

It continues Choosing a vector database: FAISS, pgvector, Qdrant and HNSW indexes (the vector store) and Embeddings and semantic search: sentence-transformers and FAISS (turning text into vectors). The model-calling part reuses the approach from Calling LLM APIs from Python: streaming, structured output and cost control. Environment: Apple M4, 16 GB RAM, Python 3.12, the openai SDK pointed at Ollama through its OpenAI-compatible endpoint http://localhost:11434/v1; generation model qwen2.5:1.5b, embedding model nomic-embed-text. The code lives in code/s7-37.

What problem RAG solves and which steps it has

The original paper by Lewis et al. (NeurIPS 2020) named the idea: combine parametric memory (knowledge stored in the model weights) with non-parametric memory (a vector index you can look up). RAG has two practical benefits: knowledge stays current without retraining, and users get sources they can check.

A basic RAG pipeline has two phases, and both run on a laptop. The indexing phase (run once, or when documents change): read the documents, split them into chunks, compute embeddings, store them. The answering phase (per question): embed the question, fetch the top-k nearest chunks, put them into the prompt, call the model, check the cited sources.

The most important rule when building RAG: measure retrieval and generation separately. If the right passage never reaches the prompt, no prompt can save you; if the right passage is there and the answer is still wrong, the fault is in the model or the prompt. The last part of this article has a table that separates these two kinds of failure.

Data and test questions for RAG

Every RAG system starts with data and a set of questions to measure it. The data/ folder holds 11 Markdown files, 16,023 characters in total, which I wrote for a fictional home-appliance store: returns and refunds, warranty, shipping, payment, guides for the air fryer, blender and rice cooker, accounts and reward points, promotions, data privacy, contact. There is no real customer data. Each file splits into sections with ## headings, a structure we will exploit when chunking.

The test set questions.json has 25 questions: 22 with an answer in the corpus (each lists the file, the section holding the answer, and the "facts" such as 30,000 or 7 days that a correct answer must contain) and 3 outside the corpus (asking about an iPhone, current prices, hiring). One question is worth 4.5 percentage points, so this RAG test set is only good for relative comparisons.

Chunking for RAG: cut by structure, not by characters

Chunking decides the unit that RAG searches and cites. The simplest approach is to cut every 400 characters. It is fast, but it has no idea where a sentence ends. Here are two consecutive chunks that fixed cutting produced from the returns file (real output of chunk_stats.py):

[doi-tra-hoan-tien.md#f0] ...'ithin 30 days of the delivery date. The store'
[doi-tra-hoan-tien.md#f1] 'covers all two-way shipping costs within t'...

The subject "The store" sits in one chunk and the predicate "covers all two-way shipping costs" in the other. The embedding of each half drifts in meaning. The second approach respects structure: each H2 section is one chunk, a section longer than max_chars is split further by paragraph, and the document title and section heading are prepended to the chunk so it can stand alone once pulled out of the document:

def chunk_by_section(docs, max_chars=900):
    out = []
    for src, text in docs.items():
        title = re.match(r"# (.+)", text).group(1)
        for heading, body in split_sections(text):
            buf = ""
            for p in (p.strip() for p in body.split("\n\n") if p.strip()):
                if buf and len(buf) + len(p) > max_chars:      # long section: split by paragraph
                    out.append(Chunk(f"{src}#{len(out)}", src, heading, f"{title} > {heading}\n{buf}"))
                    buf = ""
                buf = (buf + "\n\n" + p).strip()
            if buf:
                out.append(Chunk(f"{src}#{len(out)}", src, heading, f"{title} > {heading}\n{buf}"))
    return out

Statistics for the three strategies on the same corpus:

fixed400        48 chunks | chars: min 7, median 399, max 400
fixed400+ov80   57 chunks | chars: min 7, median 400, max 400
by-section      51 chunks | chars: min 221, median 330, max 584

The 7-character chunk in the first two strategies is a file tail that means nothing on its own; section-based cutting produces no junk chunks.

Embedding and retrieval: two details that quietly break RAG

The retrieval phase of RAG uses nomic-embed-text through Ollama's /v1/embeddings endpoint (the OpenAI compatibility docs list input as an array and dimensions among the supported parameters). This model requires task prefixes: search_document: for indexed passages and search_query: for questions. Forgetting the prefix raises no error; retrieval quality just drops silently, so I wrapped it in a single function:

def embed(texts, prefix, batch=32):
    vecs = []
    for i in range(0, len(texts), batch):
        r = client.embeddings.create(model=EMBED_MODEL, input=[prefix + t for t in texts[i:i + batch]])
        vecs += [d.embedding for d in r.data]
    v = np.asarray(vecs, dtype=np.float32)
    return v / np.linalg.norm(v, axis=1, keepdims=True)   # normalize: dot product = cosine

The Index class in rag_core.py just stores the vector matrix and computes vecs @ embed_query(q). With 51 chunks, exhaustive search with one matrix multiplication is enough and exactly accurate. Only when the corpus grows to hundreds of thousands of chunks do you need an approximate index such as HNSW.

Now measure which chunking strategy puts the right passage higher. A chunk counts as a "hit" if it has the right file and the right section holding the answer (for fixed chunks, if it contains all the facts of the question). Real output of eval_retrieval.py on the 22 questions:

strategy       chunks  hit@1  hit@3    MRR
fixed400           48   0.50   0.68  0.604
fixed400+ov80      57   0.36   0.64  0.532
by-section         51   0.64   0.82  0.742

Section-based cutting reaches hit@3 = 0.82 (18 of 22 questions have the right passage in the top 3), versus 0.68 for fixed cutting. Adding an 80-character overlap does not help and even drops hit@1 from 0.50 to 0.36. I assumed near-duplicate chunks were crowding each other out, but diag_overlap.py shows that is not the cause: the cut boundaries shift, so for many questions the chunk with all the facts no longer ranks first and gives way to an off-topic chunk (the question about exchanging a new device fell from rank 1 to rank 7). Re-running with one common criterion (right file and contains all facts) for all three strategies leaves the table above unchanged. With 22 questions, the hit@1 gap between 0.64 and 0.50 is only 3 questions, so do not draw strong conclusions.

Generating answers with citations in RAG

The generation phase of RAG is where a small model drifts most easily. The first attempt, free_prompt.py, asked the model to "write the passage number like [1] after each claim" in free-form text. With qwen2.5:1.5b, the result on the 6 questions of rag_demo.py (output in free_prompt_output.txt): 3 answers had no source number in the form [n], and 3 answers were only a reference like [1] with no content (for example, the refund question returned [2] (bao-mat-du-lieu.md / Data collected)). Small models follow free-form format instructions poorly. A more robust way is to make the model return JSON with two fields, answer and sources, and then validate in code instead of trusting it:

def answer(question, index, k=3, model=None):
    hits = index.search(question, k)
    r = client.chat.completions.create(
        model=model or CHAT_MODEL, temperature=0, seed=7, max_tokens=300,
        response_format={"type": "json_object"},
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user", "content": build_prompt(question, hits)}])
    raw = r.choices[0].message.content
    try:
        data = json.loads(raw)
        text = str(data.get("answer", "")).strip() or NOT_FOUND
        src = [int(x) for x in data.get("sources", []) if str(x).isdigit()]
    except (json.JSONDecodeError, TypeError, ValueError):
        return raw.strip(), hits, []                  # broken JSON: no sources
    cited = sorted({n for n in src if 1 <= n <= len(hits)})   # drop passage numbers that do not exist
    return text, hits, cited

The system prompt gives three instructions: use only the numbered passages, return JSON in the exact shape, and if the information is missing, reply with exactly "I could not find this information in the documents." and an empty sources; the end of the prompt has one worked example (few-shot) because a 1.5-billion-parameter model needs to see the expected output shape (see Systematic prompt engineering: roles, examples and prompt testing). Set temperature=0 and seed for repeatability; according to the Ollama API docs, reproducible output requires setting seed together with temperature at 0. Real output of rag_demo.py (two questions, abridged):

Q: What does error E03 mean on the air fryer?
A: Error E03: overheating because the basket is misplaced or full of grease; turn the unit off, let it cool for 30 minutes, then reseat the basket.
   top-3: ['noi-chien-khong-dau/Error codes 0.75', 'noi-com-dien/Common problems 0.72', 'noi-com-dien/Cleaning and replacement 0.71']
   cited: ['noi-chien-khong-dau.md / Error codes'] | 2.0s

Q: How many days is the store closed for the New Year holiday?
A: I could not find this information in the documents.
   top-3: ['cua-hang-lien-he/Experience store 0.75', 'bao-mat-du-lieu/Purposes of use and sharing 0.74', 'doi-tra-hoan-tien/Return fees 0.73']
   cited: none | 1.1s

The second question is a useful failure: the documents do have the holiday information (the "Working hours" section of the contact file), but retrieval did not put that section in the top 3, so the model correctly refused as instructed. The refusal behavior is correct; the bug is in retrieval.

Checking RAG: separating retrieval errors from generation errors

answer_check.py runs all 22 answerable questions and records three columns: whether the right passage is in the top 3, whether the answer contains all the facts, and whether the source the model cited hits the right section. Summary:

retrieval top-3 correct : 18/22
answer contains facts   : 16/22
  when retrieval correct: 16/18
  when retrieval missed : 0/4
citation hits section   : 17/22

The key numbers are lines 3 and 4: when retrieval missed, 0 of 4 answers were correct. The upper bound of the whole system is retrieval quality (18/22 = 82%); the generation model only lost 2 more questions. To improve this RAG system, investing in retrieval pays off more than swapping the generation model. The four retrieval misses (fried chicken wings, blender running continuously, redeeming reward points, New Year holiday) share one trait: they use words that rarely overlap with the text of the section holding the answer, exactly the kind of failure that the next article (hybrid search and reranking) targets.

The remaining three cases deserve a close read (see failures_output.txt and rag_demo_output.txt).

For the question "What is the maximum COD order value?", the right section is at rank 1, but the model answered "from 3,000,000 VND upward needs confirmation...", picking the wrong number from the same chunk: a genuine generation error. For the question "Is the warranty valid if water gets into the motor?", the answer "No" is correct in substance but too terse, so the fact matcher marked it wrong; that is a limit of substring matching, and the reason an LLM-as-judge is needed in the evaluation article.

For the question "How long does a refund to a bank card take?", the model answered "7 to 10 days" correctly, but the cited source pointed to the data privacy chunk, so a citation can be wrong even when the answer is right. That is why the model's citations are only hints; a real product should always display the full top-k so users can cross-check.

Out-of-corpus questions: a score threshold is not enough

The common reflex is to set a threshold: if the top-1 cosine score is below X, refuse without calling the model. refuse_check.py measures whether a threshold is feasible:

top-1 cosine, 22 in-corpus questions : min 0.730, median 0.784, max 0.839
top-1 cosine, 3 out-of-corpus questions: 0.762, 0.793, 0.812

refused correctly 3/3

The two score ranges overlap: out-of-corpus questions reach 0.812, while 15 of the 22 valid questions score below 0.80. A threshold of 0.80 would wrongly refuse 15 valid questions and still let one of the three out-of-corpus questions through. With this embedding model and a small corpus, the similarity score only says "close to the store's topic", not "has an answer". What saves the system here is the refusal instruction in the prompt: all 3 out-of-corpus questions were refused correctly. That is only 3 questions; do not treat 3/3 as a real refusal rate, and put a few dozen out-of-corpus questions in your test set.

RAG variants and limits

• Chunk size and k. Small chunks retrieve precisely but lack context; large chunks have enough context but diluted embeddings. A larger k raises the chance of including the right passage, but a longer prompt costs more tokens, and the Lost in the Middle study shows models use information in the middle of a long context worse than at the start and end.
• Hybrid retrieval and reranking. For RAG, combine keywords (BM25) with vectors, then rescore the top candidates with a cross-encoder; most serious RAG systems have this step. It is the topic of the next article.
• Parent-document retrieval and HyDE. Search with small chunks but put the whole section or parent document into the prompt; or have the model write a hypothetical answer and embed that instead of the question (HyDE). This article did not measure either approach.

When not to use RAG

RAG fits textual knowledge that changes and needs citations.

RAG does not fit when the answer requires computation over structured data (this quarter's revenue needs SQL, not embeddings), when the documents are so few that they fit in one prompt (putting the whole corpus in the context is simpler and more accurate; this article's 16-thousand-character corpus is only a few thousand tokens), or when the goal is to change the model's style or tone (that is a job for prompting or fine-tuning).

Also do not treat RAG as a security measure: any document put into the prompt can be leaked to the asker, so access control must happen at the retrieval step, and instructions hidden in embedded documents can steer the model (indirect prompt injection).

Common mistakes when building RAG

1. Forgetting the embedding prefix: with models that require prefixes like nomic, omitting them makes retrieval worse without any error.
2. RAG chunks that lose context: a passage saying "a fee of 30,000 VND" does not say which fee. Prepend the document and section titles to the chunk.
3. Trusting the similarity score as a refusal threshold without measuring the score distribution of in-corpus and out-of-corpus questions as shown above.
4. Trusting the model's citations: validate the passage numbers in code, and display the real sources.
5. Evaluating only the final RAG answer: you cannot tell whether retrieval or generation failed, so you fix the wrong place.

RAG checklist

• For every RAG project, cut chunks by document structure, attach headings to chunks, and measure hit@k and MRR before choosing a size.
• Normalize embeddings, use the right prefix, use the same model for both phases, and cache by content hash.
• Make the model return JSON with answer and sources, and validate the sources in code.
• Always include out-of-corpus questions in the test set and a clear refusal rule in the prompt.
• Report RAG results separately: how often retrieval is right, and when retrieval is right, how often generation is right.

The source code of this RAG article includes rag_core.py (chunking, embedding, retrieval, generation), chunk_stats.py, eval_retrieval.py, rag_demo.py, refuse_check.py, answer_check.py, failures.py, free_prompt.py, diag_overlap.py together with questions.json, data/ and the *_output.txt files. Every number in this article is real output on an Apple M4 with 16 GB, using 25 self-written questions over a hypothetical document set; the numbers will change if you change the documents, the model or the questions.

Next article

Advanced RAG: hybrid search, reranking and query rewriting. The four retrieval misses from today are the place to experiment: the next article adds BM25, merges results with Reciprocal Rank Fusion and reranks with a cross-encoder, then measures how much each step is worth.