The previous article ended on an uncomfortable number: out of 22 questions, 4 never got the right passage into the top 3. When the right passage is not in the prompt, the model makes things up or refuses, and no prompt can save it. This article continues with retrieval and covers the three techniques most often mentioned under "advanced RAG": hybrid search (BM25 combined with vectors), cross-encoder reranking, and query rewriting. What sets it apart is that all three are measured on the same 48 questions, and the results are not as pretty as the marketing: some techniques help, some make things worse, and the reason depends on which embedding you use.

It reuses the document set and chunking function from Building RAG in Python: chunking, retrieval and cited answers, and the vector store and embeddings from Choosing a vector database: FAISS, pgvector, Qdrant and HNSW indexes and Embeddings and semantic search: sentence-transformers and FAISS. Environment: Apple M4, 16 GB RAM, Python 3.12, CPU only, each process under 1.5 GB RAM; the small LLM qwen2.5:1.5b and the nomic-embed-text embedding run through Ollama (OpenAI-compatible endpoint http://localhost:11434/v1, no real key needed). The documents and questions are in Vietnamese; in this article the questions and section titles shown in outputs are translated into English.

Why vector top-k is not enough: two different kinds of misses

Vector retrieval finds passages that are close in meaning but not required to contain the right words; keyword retrieval is the opposite. That is why people combine them, then add reranking afterward to reorder the candidate pool. This split of roles matches the Retrieve & Re-Rank documentation of Sentence Transformers: a bi-encoder (embedding) encodes passages and questions independently, so passage vectors can be precomputed and search is fast on a large corpus; a cross-encoder feeds the question and the passage into the model together, which is more accurate but slower, so it is used only on a small candidate pool.

The benchmark: 48 questions in three difficulty groups

The corpus is still 11 Markdown files about a fictional home-appliance store, chunked by H2 section into 51 chunks. The question set grows from 22 to 48 questions in three groups: A (the 22 original questions from the RAG article, close to the wording of the documents), B (14 everyday questions that still contain keywords from the documents), and C (12 questions that deliberately avoid the documents' wording, such as "how many times will the courier come back if I'm not home" instead of "failed delivery").

To be clear: groups B and C were written by me after reading the documents, 48 questions is a small set (each question is worth 2.1 percentage points), and with only 51 chunks a "top 10" already covers 20% of the corpus. Differences under about 4 points are not enough to conclude anything; the numbers are for relative comparison, not for promising a level of accuracy. The metrics for reranking are hit@k (the right passage is among the first k results) and MRR (mean reciprocal rank of the right passage).

Step 1: BM25 and hybrid search with Reciprocal Rank Fusion

BM25 needs a tokenizer. Vietnamese is written with spaces between syllables, so the simple approach is to lowercase, keep the diacritics and split on word characters. The rank-bm25 library does the scoring:

def tokenize(text):
    """Split by syllable: lowercase, keep Vietnamese diacritics, drop punctuation."""
    return re.findall(r"\w+", unicodedata.normalize("NFC", text).lower())

self.bm25 = BM25Okapi([tokenize(c.text) for c in self.chunks])
def sparse(self, q):
    return list(np.argsort(-self.bm25.get_scores(tokenize(q))))

The two rankings (BM25 and vector) have scores on scales that cannot be compared: a BM25 score may be 9.2 while cosine similarity sits around 0.8. Reciprocal Rank Fusion ignores the scores and uses only the ranks. The formula comes from Cormack, Clarke and Büttcher (SIGIR 2009): the score of a passage is the sum of 1/(k + rank) over every list it appears in. The constant k reduces the weight of the top ranks; 60 is the common default.

def rrf(rankings, k=60):
    score = {}
    for r in rankings:
        for pos, idx in enumerate(r, 1):
            score[idx] = score.get(idx, 0.0) + 1.0 / (k + pos)
    return sorted(score, key=lambda i: -score[i])

The big advantage of RRF is that there are no weights to tune: adding a ranking source means adding one more list to rankings. The drawback: it treats the two sources as equals, so when one source is much weaker, it drags the result down. The measurements below show exactly that.

Step 2: reranking with a cross-encoder

A cross-encoder takes a (question, passage) pair and returns a relevance score. Because the model has to run once per pair, we rescore only the first n_cand candidates from the retrieval step and keep the rest in their old order.

The model used here is cross-encoder/mmarco-mMiniLMv2-L12-H384-v1: a multilingual MiniLMv2 base trained on mMARCO (Apache-2.0 license, about 470 MB of weights). mMARCO is a machine translation of MS MARCO into many languages, and the mMARCO paper by Bonifacio et al. and the dataset page list Vietnamese among the languages. Note: the data is machine-translated English web queries, not questions about store policies, so you must measure on your own data.

def rerank(self, q, order, n_cand=10):
    cand, rest = list(order[:n_cand]), list(order[n_cand:])
    scores = self.ce.predict([(q, self.chunks[i].text) for i in cand], batch_size=16)
    return [cand[j] for j in np.argsort(-scores)] + rest

The cross-encoder score here is a raw score (before the sigmoid), and it only means something when comparing passages for the same question; do not set a fixed threshold and compare across different questions without calibrating.

Results: the embedding decides, reranking rescues the rest

Run eval_all.py (with torch.set_num_threads(2) and device="cpu"). This is the real output:

48 questions: A = 22 original RAG-article questions, B = 14 everyday questions (still use document words), C = 12 questions that deliberately avoid document words
ranking method                 hit@1 hit@3 hit@5   MRR |  A@3  B@3  C@3 | ms/question
BM25 (keyword)                  0.75  0.88  0.90 0.814 | 1.00 0.93 0.58 |      0
dense nomic-embed-text          0.33  0.46  0.65 0.465 | 0.82 0.14 0.17 |     31
dense multilingual-e5-small     0.88  0.98  0.98 0.922 | 1.00 1.00 0.92 |     26
hybrid RRF: nomic + BM25        0.52  0.79  0.83 0.663 | 1.00 0.86 0.33 |     34
hybrid RRF: e5 + BM25           0.81  0.94  0.94 0.878 | 1.00 1.00 0.75 |     16
rerank top10 of nomic+BM25      0.83  0.88  0.88 0.857 | 1.00 0.93 0.58 |    236
rerank top10 of e5+BM25         0.88  0.96  0.98 0.916 | 1.00 1.00 0.83 |    231

Read this table in layers. First, nomic-embed-text reaches only 0.46 hit@3, and nearly collapses on groups B and C (0.14 and 0.17). The model card of nomic-embed-text-v1 lists the language as English; that is a reasonable explanation for the result, although I did not prove it with a separate experiment. In the previous RAG article, 18/22 correct (group A, 0.82) looked fine because the questions used the documents' own words. The weakness only shows when the question is phrased differently.

Second, BM25 alone, with no model at all, already reaches 0.88 hit@3, clearly better than an English embedding. Third, combining nomic with BM25 through RRF (0.79) is worse than BM25 alone: the weak source adds noise to the strong one. Hybrid search is not magic, and neither is reranking; they only help when both branches have value. Fourth, reranking rescues a lot: from the nomic hybrid (hit@1 0.52) up to 0.83, which is on par with BM25 but with a higher MRR (0.857 versus 0.814).

Once the embedding is good, are hybrid and reranking still worth it

Try replacing nomic with multilingual-e5-small (the report by Wang et al.: multilingual, three sizes; the small version has 12 layers, 384-dimensional vectors and about 470 MB). The prefix query: is required for questions and passage: for passages. The real result of dense_multilingual.py:

multilingual-e5-small: hit@1 0.88 hit@3 0.98 hit@5 0.98 MRR 0.922 | A 1.00 B 1.00 C 0.92

A small model running on CPU, 26 ms per question, reaches 0.98 hit@3 and 0.922 MRR. Hybrid with BM25 does not improve it: hit@3 drops to 0.94 and MRR to 0.878, because BM25 is weak on group C and pulls the result down. Reranking the top 10 brings hit@3 back to 0.96 and MRR to 0.916 (close to e5 alone at 0.922), and does not even surpass it. Hit@1 is 0.88 in both.

That means for reranking on this small corpus, with a good multilingual embedding, adding BM25 and reranking gives no measurable improvement; on hit@3, 0.98 versus 0.96 is a single question out of 48, and MRR 0.922 versus 0.916 is within noise. What I take from this, unlike many introductory articles: before adding a layer, measure the simple retriever with an embedding that fits your language. Hybrid search and reranking are insurance for the group of questions where one branch fails, not a default upgrade. They are worth more when the corpus is larger, the questions contain product codes and proper names, or you cannot change the embedding.

One question through every layer

trace_one.py prints the top 4 of each layer for the question "How long can I use my appliance before the manufacturer's commitment runs out?" (the answer is the section "Warranty period by product group" of bao-hanh.md). Real output, shortened to the e5, RRF and reranking layers:

e5   top-4: doi-tra-hoan-tien / Return window; bao-hanh / Process and handling time; bao-hanh / Warranty period by product group; ...
RRF  top-4: van-chuyen / Failed delivery; doi-tra-hoan-tien / Non-returnable products; khuyen-mai / Order cancellation and vouchers; ...
rerank top-4 (cross-encoder score, before sigmoid):
     -3.98  doi-tra-hoan-tien / Return window
     -4.57  bao-hanh / Warranty period by product group

BM25 was fooled by common words that appear everywhere in the documents (the equivalents of "can", "how", "long"); the right passage sits at rank 23. e5 puts the right passage at rank 3, but RRF pushes it down to rank 7 because BM25 votes for other passages. Reranking lifts it to rank 2, behind the "Return window" passage, which is also about time limits; the first two passages are 0.59 raw points apart, a thin margin rather than a firm verdict.

How many candidates to rerank

n_cand is the dial between quality and latency. Real output of sweep_ncand.py on the e5 + BM25 hybrid (latency fluctuates because the machine was running other work):

hybrid, no rerank: hit@1 0.81 hit@3 0.94 MRR 0.878
n_cand hit@1 hit@3   MRR  ms/question   answer inside the first n_cand of the hybrid
     3  0.92  0.94 0.930      53   45/48
     5  0.88  0.94 0.909      78   45/48
    10  0.88  0.96 0.916     153   47/48
    20  0.88  0.96 0.910     296   47/48

Latency grows almost linearly with n_cand, while quality is flat beyond roughly 3 to 10: the answer is already within the first 3 and the first 5 candidates for 45/48 questions, rising to 47/48 from n_cand 10. The 0.92 hit@1 at n_cand=3 does not mean "3 is best"; a difference of one or two questions is within noise. A practical rule for reranking: choose n_cand so that the "answer is in the pool" rate is close to its maximum (here 10), because reranking cannot rescue a passage that is not in the pool.

The fusion_k.py experiment confirms two things: changing the RRF k from 1 to 200 barely changes the result (hit@3 goes from 0.94 to 0.92 at k=200), so do not spend effort tuning k at this scale; and reranking the top 10 of BM25 alone gives only 0.92 hit@10 because for 4 questions BM25 does not put the answer in the top 10, and a cross-encoder cannot do anything about them.

Step 3: query rewriting, the rival of reranking

There are two common ideas. Query rewriting (rewrite-retrieve-read), as in Ma et al., "Query Rewriting for Retrieval-Augmented Large Language Models", adds an LLM step that turns the raw question into a better query. HyDE, in Gao, Ma, Lin and Callan, asks the LLM to write a hypothetical answer passage (which may be wrong in its details) and then uses that passage to search.

Here both use qwen2.5:1.5b through Ollama with temperature=0; each call takes about 1.1 seconds, 7 times the cost of reranking the top 10. Real output of rewrite.py, only the rows for hybrid e5 + BM25 and reranking:

retrieval method                           hit@1 hit@3   MRR |  A@3  B@3  C@3
hybrid e5+BM25, original question           0.81  0.94 0.878 | 1.00 1.00 0.75
hybrid e5+BM25, rewritten question          0.75  0.85 0.814 | 0.91 0.93 0.67
hybrid e5+BM25, original + rewritten        0.77  0.92 0.852 | 0.95 1.00 0.75
hybrid e5+BM25, HyDE                        0.83  0.92 0.887 | 1.00 0.93 0.75
hybrid e5+BM25, original + rewritten + HyDE 0.88  0.96 0.916 | 1.00 1.00 0.83
rerank top10 (queried with original), original 0.88  0.96 0.916 | 1.00 1.00 0.83

The rewritten question makes things worse when it replaces the original (hit@3 from 0.94 down to 0.85), even when mixed with the original (0.92). Only the original + rewritten + HyDE combination reaches 0.96, equal to reranking but costing two LLM calls (about 2 seconds) versus 0.15 seconds.

The reason shows up when reading the rewrites for group C: for "I order late in the afternoon, when do you start preparing it?", the model answered "We start preparing goods early in the morning..." (a made-up answer, not a rewrite); for the courier question, it copied the question verbatim with the prefix "Rewritten question:". A 1.5-billion-parameter model is too small for this job.

I did not filter the prefix in the measurement code, so this is a lower bound; a prompt with few-shot examples may improve it but would not change the conclusion about cost. Query rewriting is worth trying with multi-turn conversations ("what about the other one?") and a model strong enough; for single-turn questions on a small corpus, it is not yet worth it.

Questions that still miss: the real limits of reranking

failures.py lists the questions where hybrid misses the top 3 or reranking does not bring the answer to rank 1 (columns: BM25, e5, hybrid, reranking). Real output, three representative rows:

Can I still return it if I bought it and got bored of it?    6   1   2   3
Is there any extra charge for paying the courier in cash?   41  13  22  22
The blender jar leaks from the bottom, what should I do?      1   1   1   2

In the first and last questions, e5 already had the right passage at rank 1, but reranking dropped it to rank 3 and 2: reranking can also break a result that was already correct. The courier question is the worst: the documents say "COD" and "surcharge", while the asker writes "cash" and "courier", and no branch can connect them. The fix is not in the ranking but in the data: add synonyms to the documents ("COD, cash payment to the delivery person") or read the logs of real questions.

When to add a layer and when not to

  • Start by measuring: the right-language embedding first, then BM25, then reranking; keep each layer only if the benchmark shows it helps.
  • Hybrid fits when the content has error codes, product codes and proper names (like "Code E05") that embeddings tend to blur. BM25 is also the default ranker of Elasticsearch (see Elasticsearch on Kubernetes if you already have a cluster).
  • Reranking fits when the candidate pool contains the answer but in the wrong order, the corpus is larger, and you accept an extra 50 to 300 ms per question on CPU.
  • Query rewriting only for multi-turn conversations or long, ambiguous questions, with a strong enough model; always keep the original question as one source.

Common mistakes

  • Forgetting the query: and passage: prefixes of the E5 family: quality drops silently, with no error.
  • Reranking the whole corpus: cost grows linearly with the number of pairs.
  • Setting a "no answer" threshold on the raw cross-encoder score without calibrating: scores have no common scale across questions.
  • Using an English embedding for Vietnamese documents and treating hybrid search as the cure: as the table shows, it only brings the result back to BM25 level.

Checklist for reranking and hybrid retrieval

  • Measure BM25, dense and hybrid separately before combining them; record hit@k and MRR by easy and hard question group.
  • Choose a multilingual embedding for Vietnamese documents, and verify it with questions that avoid the documents' wording.
  • Use RRF with k = 60 as a starting point, then add reranking at the end with an n_cand such that the answer is almost always in the pool.
  • Add query rewriting only for a concrete reason; fix at the data layer the questions that ranking cannot save.

The source code consists of rag_core.py (reused from the RAG article), hybrid.py, eval_all.py, dense_multilingual.py, sweep_ncand.py, fusion_k.py, failures.py, trace_one.py, rewrite.py, questions.json, data/, requirements.txt and the *_output.txt files. The library versions in requirements.txt are the ones used when re-running and checking the article, and may be newer than the publication date. The numbers are real output on an Apple M4, CPU; latency may vary when the machine is running other work.

Next article

RAG evaluation: gold question sets, retrieval accuracy and faithfulness. This article already touched the limits of a hand-written 48-question set; the next one builds a more trustworthy benchmark and separates retrieval accuracy from the faithfulness of the answer.