The search box on most websites still matches keywords: a customer types "drop my order" (the colloquial Vietnamese phrase), the system finds no such word in the article "How to cancel an order," and returns an empty page. Semantic search (search by meaning) fixes exactly this: text is turned into vectors so that sentences with the same meaning sit close together, and the search runs on distance. This article builds a set of 48 online-shop FAQ entries and 24 questions that are deliberately worded with different vocabulary, measures BM25 against two embedding models, measures the query:/passage: prefixes, tries a cross-encoder for reranking, then compares the FAISS index types (Flat, IVF, HNSW, IVF-PQ) on 200,000 vectors.

The article follows Running a local LLM with Ollama and llama.cpp: model choice, quantization and performance and Word embeddings: Word2Vec, FastText and representing word meaning, and uses the measurement approach from Model evaluation for classifiers: precision, recall, ROC and imbalanced data. Environment: Apple M4, 16 GB, Python 3.12, sentence-transformers 3.3.1, torch 2.5.1, faiss-cpu 1.9.0.post1, transformers 4.46.3. Every number is real output; the code is in code/s7-35. The FAQ data and questions in the experiment are in Vietnamese; this article shows English translations of them, so the vocabulary mismatches described below are paraphrased rather than reproduced word for word.

Why keyword search is not enough

Before semantic search, keyword search (TF-IDF, BM25) compared the words two texts have in common. It is fast, explainable, and very strong when users type the exact term, such as a product code or a proper name. But everyday Vietnamese is full of synonyms and alternative phrasings: the English loanword "ship" instead of the formal word for delivery, "pass" instead of the formal word for password, "red invoice" (the colloquial name) instead of "VAT invoice," "drop the order" instead of "cancel the order." Keyword matching misses exactly these queries, and that is the gap semantic search fills.

Semantic search flips the view: an embedding model trained on a large body of text maps each sentence to a vector, usually a few hundred dimensions. Two sentences with the same meaning produce two vectors that are close under cosine similarity, even when they share no words. The E5 paper (Wang et al., 12/2022) is one example: a model trained contrastively on weakly supervised text pairs that, according to its authors, is the first to beat BM25 on BEIR in the zero-shot setting. But "beat" here is an average over many English datasets; on your own Vietnamese data you have to measure the result yourself, as the sections below do.

How semantic search works: from text to vector

A semantic search pipeline has two phases. In the indexing phase, you embed every document into a vector, normalize it to length 1 and store it in a search structure. In the query phase, you embed the question with the same model, find the k nearest vectors and return the matching documents. For semantic search, the Sentence Transformers documentation distinguishes symmetric search (query and document have similar length, as when finding duplicate questions) from asymmetric search (a short query against longer passages, which is exactly this FAQ case) and stresses choosing a model built for the right kind of task.

On the math of semantic search: if all vectors are L2-normalized, cosine equals the inner product, and ranking by L2 distance gives the same order as ranking by cosine. The first part of faiss_index.py checks this on 1,000 random vectors:

cosine == IP: True | cosine == L2 (normalized): True

The practical consequence for semantic search: always pass normalize_embeddings=True when embedding, then use an inner-product index (IndexFlatIP). Forgetting to normalize is the most common semantic search bug, because the "inner product" then depends on vector length as well as direction.

A minimal semantic search with sentence-transformers and FAISS

The model used for semantic search here is intfloat/multilingual-e5-small, from the same family as the Multilingual E5 Text Embeddings paper (2/2024): three sizes (small, base, large) trained in two stages (contrastive pre-training on about 1 billion multilingual text pairs, then fine-tuning on labeled data). The small version produces 384-dimensional vectors and is light enough to run on a CPU. The sample data in faq_data.py is 48 hand-written FAQ entries (about 20 words on average) on returns, shipping, payment, warranty and products; it contains no real customer data. The minimal semantic search snippet (demo_search.py):

from sentence_transformers import SentenceTransformer
import numpy as np, faiss
from faq_data import DOCS

model = SentenceTransformer("intfloat/multilingual-e5-small", device="cpu")
D = model.encode(["passage: " + d for d in DOCS], normalize_embeddings=True)   # (48, 384)
index = faiss.IndexIDMap2(faiss.IndexFlatIP(D.shape[1]))                      # IP on normalized vectors = cosine
index.add_with_ids(D, np.arange(len(DOCS)).astype("int64"))
faiss.write_index(index, "faq.faiss"); index = faiss.read_index("faq.faiss")

# the real query is in Vietnamese; here it is translated
q = model.encode(["query: If my device falls on the ground and the screen breaks, is it covered by warranty?"], normalize_embeddings=True)
scores, ids = index.search(q, 3)

Real semantic search output for three questions, each worded very differently from the FAQ entries (questions and entries translated from Vietnamese):

index: 48 vectors, d=384

Question: I'm 92 cm around the chest, which size should I wear?   (8.1 ms including query embedding)
  0.867  [18] How to choose a shirt size: measure your chest and compare it with the size chart, if you are between two siz
  0.820  [27] A 14-inch office laptop weighs 1.3 kg, suitable for carrying to school and to work every day.
  0.820  [21] Cotton T-shirts may shrink slightly after the first wash, so pick a slightly looser fit if

Question: If my device falls on the ground and the screen breaks, is it covered by warranty?   (6.3 ms including query embedding)
  0.878  [16] Warranty does not cover devices that are dropped, damaged by water or have had their hardware tampered with
  0.842  [17] Laptop batteries have a 6-month warranty, all other components 24 months.
  0.839  [15] Genuine phone warranty is 12 months, manufacturer defects are replaced with a new unit within the first

Question: How much memory do I need for 4K video editing?   (6.1 ms including query embedding)
  0.892  [30] 16 GB of RAM is enough for programming and photo editing, choose 32 GB if you edit 4K video.
  0.825  [28] A gaming laptop needs a discrete graphics card and large cooling fans, so it is heavy
  0.824  [25] The phone supports two nano SIMs and one microSD card slot up to 1 TB.

"Screen breaks" matches "dropped or damaged," "memory" matches "RAM," "92 cm chest" matches "measure your chest": exactly what BM25 cannot do. But look closely at the scores: the correct results reach 0.87-0.89, while the wrong ones still score 0.82-0.84. The cosine scores of the E5 family are squeezed into a narrow band (the model card's FAQ says cosine scores fall roughly between 0.7 and 1.0 because of the low temperature of 0.01 used in training, and advises comparing only the relative order of scores), so you cannot set a "above 0.8 means relevant" threshold for this model. The fourth question I tried is one with no answer in the FAQ:

Question: We have a cat at home, should we buy a vacuum cleaner?   (6.2 ms including query embedding)
  0.833  [20] Machine-wash this product on a gentle cycle, do not use strong detergent and do not dry at high heat
  0.827  [39] To review a product, you need to be logged in and to have bought that product within the last 6 mon
  0.821  [44] In-home installation applies to air conditioners and TVs over 55 inches, fees from 300.

The top result scores 0.833, nearly the same as the wrong second result for the "screen breaks" question (0.842), even though its content is completely unrelated. A real semantic search system needs a way to handle "no answer": a threshold calibrated on your own data, or a second check such as the cross-encoder in a later section.

Measuring semantic search: BM25 versus embeddings on 24 questions

To know whether semantic search is worth it, you have to measure. search_eval.py scores 24 questions, each with exactly one correct FAQ entry, using three metrics: Recall@1 (the answer is ranked first), Recall@3, and MRR (the mean reciprocal rank of the correct answer). BM25 is a hand-written implementation that tokenizes by syllable. I assigned the answer labels by hand; the first run gave unusually low results because the labels of five questions were off by one position, so I fixed the labels and reran everything. Errors in evaluation data are more likely than errors in the model.

method                                  R@1    R@3    MRR
BM25 (keyword)                         0.58   0.71  0.677
e5-small, with prefixes                0.62   0.83  0.759
e5-small, without prefixes             0.67   0.92  0.798
nomic-embed-text, with prefixes        0.38   0.50  0.490
nomic-embed-text, without prefixes     0.33   0.46  0.444

Read each row carefully, because there are only 24 questions (each one moves the result by 4.2 percentage points):

  • e5-small beats BM25 on every metric (R@3 0.83 versus 0.71, which is 20 questions versus 17 in the top 3), but the gap at R@1 is only 15 questions versus 14. BM25 is not bad, because the questions, though worded differently, still share many words. Semantic search wins most clearly in the top 3, that is, when you hand a few candidates to a later step.
  • The query:/passage: prefixes: the E5 model card says plainly "this is how the model is trained, otherwise you will see a performance degradation" and recommends this pair of prefixes for asymmetric retrieval. On this set, running without prefixes came out slightly ahead (R@3 0.92 versus 0.83), but that difference is 2 questions out of 24, not enough to conclude that you should drop the prefixes. I still use the prefixes in the main configuration and only note that this is something to re-measure on your own data.
  • nomic-embed-text is clearly weaker (R@1 0.38, worse than BM25). The reason is easy to see: the Nomic Embed paper describes an English embedding model (8192-token context) trained for English. It runs through Ollama, which is convenient, but an English embedding is not the right choice for Vietnamese. This experiment exists to help you avoid that mistake.

The list of questions e5-small got wrong shows where the model fails: "Does shipping cost money if I place a big order?" (the answer is the shipping fee entry) was pulled toward the pre-order entry; "I don't remember my account pass anymore" (the answer is forgotten password) was pulled toward the data security entry; "Can I swipe a card and pay it off month by month" was pulled toward the gift entry. These are slang and shorthand ("ship," "pass," "swipe a card") that Vietnamese training data covers thinly. The practical fix for Vietnamese semantic search: add synonyms to the FAQ entries themselves, or add sample questions to each entry and embed those as well.

Improving semantic search: more models and cross-encoder reranking

Two things are worth trying before you increase the model size in semantic search. First, another multilingual embedding model with the same 384 dimensions. Second, reranking the top 5 with a cross-encoder: instead of embedding the question and document separately and comparing vectors (a bi-encoder), a cross-encoder reads the (question, document) pair together and scores relevance, which is more accurate but much slower because it must run for every pair. The model used is cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, trained on a multilingual translation of MS MARCO. Results from models_rerank.py:

model                                        dims    R@1   R@3   R@5    MRR  ms/query
multilingual-e5-small                         384   0.62  0.83  0.92  0.759     4.5
paraphrase-multilingual-MiniLM-L12-v2         384   0.62  0.75  0.75  0.713     4.6

Rerank the e5-small top 5 with cross-encoder mmarco-mMiniLMv2-L12-H384-v1:
  before rerank  R@1=0.62 R@3=0.83 MRR=0.759
  after rerank   R@1=0.75 R@3=0.92 MRR=0.838   (+46 ms/query for 5 pairs)
  questions fixed by rerank: 5, questions broken: 2

Two takeaways for semantic search. The paraphrase-multilingual-MiniLM-L12-v2 model (trained to find equivalent sentences, a symmetric problem) is worse than e5-small at R@3 and R@5 (0.75 versus 0.83 and 0.92) in this asymmetric case, just as the Sentence Transformers documentation warns about choosing a model by task. Reranking raises R@1 from 0.62 to 0.75: it fixes 5 questions but breaks 2, a net gain of 3 out of 24. That is enough to consider but not enough to be sure; the cost is about 46 ms more per query (five pairs, measured on CPU), 10 times the embedding time. Reranking always trades latency for accuracy, and it only applies to a small top-k.

FAISS for semantic search: choosing Flat, IVF, HNSW or PQ

For semantic search over 48 entries, brute force (Flat) is enough and is the only option that gives exactly correct results. The real question is when you need an approximate index (ANN). The FAISS guidelines for choosing an index summarize it: Flat gives exact results and needs no training; HNSW is very fast and accurate but uses a lot of RAM, at d*4 + M*2*4 bytes per vector, and cannot delete vectors; IVF splits the data into clusters and needs training data of about 30 to 256 times the number of clusters; PQ compresses a vector to M bytes when memory is the priority. HNSW is based on the Malkov and Yashunin paper: a multi-layer graph whose search complexity, the authors state, grows logarithmically.

To see the differences in numbers, faiss_index.py generates 200,000 synthetic 384-dimensional vectors with 400 clusters (synthetic data, not real text) and compares four index types with Flat as the reference. Recall@10 is the fraction of the approximate index's 10 results that match the 10 correct results from Flat:

data: 200,000 vectors x 384 dims = 293 MiB float32; 500 queries, k=10
index                         build s     MiB  ms/query  recall@10
Flat (brute force)                0.0     293      0.17      1.000
IVF nlist=512 nprobe=1            0.9     295      0.03      0.940
IVF nlist=512 nprobe=8            0.9     295      0.08      1.000
IVF nlist=512 nprobe=32           0.9     295      0.31      1.000
HNSW M=32 efSearch=16            27.4     345      0.06      0.889
HNSW M=32 efSearch=64            27.4     345      0.13      0.997
HNSW M=32 efSearch=128           27.4     345      0.14      1.000
IVF-PQ m=48 nprobe=32            15.6      12      0.14      0.196

reloading IVF from disk gives the same results: True

The verified analysis:

  • At 200,000 vectors, Flat already takes only 0.17 ms per query. This figure is measured on a batch of 500 queries on one thread (FAISS uses matrix multiplication for batches), so single queries will be slower; I did not measure that case. The firm conclusion: with a few hundred thousand vectors, ANN is not yet necessary, and Flat is simple and exactly correct.
  • IVF is almost free in memory (295 MiB versus 293) and builds in under 1 second; raising nprobe from 1 to 8 lifts recall from 0.94 to 1.00. Note that nprobe=32 is slower than Flat on this data (0.31 ms versus 0.17 ms): scanning many clusters on a single thread loses the advantage.
  • HNSW is fast but the build takes 27.4 seconds (30 times IVF) and costs 18% more memory (345 MiB). The measurement matches the FAISS formula: 384*4 + 32*2*4 = 1,792 bytes per vector, times 200,000 is about 342 MiB. efSearch is the recall dial: 16 gives 0.889, 64 gives 0.997.
  • IVF-PQ compresses 293 MiB to 12 MiB (24 times) but recall is only 0.196. This needs careful reading: the synthetic data mixes in Gaussian noise that is hard to compress, and I did not enable the re-ranking step with the original vectors that the FAISS guidelines recommend alongside PQ. I do not claim PQ gives this recall on real text embeddings, only that PQ costs accuracy and must be measured on your own data.

The last line checks that saving and reloading with write_index/read_index gives the same results, a prerequisite for the index to survive a service restart. For large-scale semantic search, the FAISS guidelines also recommend an IVF cluster count of 4 to 16 times the square root of the vector count when under one million; for 200,000 vectors that is about 1,800 to 7,200, while I used 512 to keep the build fast, so the IVF numbers here are not the optimal configuration. Running at scale also involves metadata filtering, updates and replication, which come in the next article.

Cost and resource limits of semantic search

The cost of semantic search at this scale is very small: embedding the 48 entries takes 0.64 seconds (loading a cached model takes about 0.5 seconds), embedding one question takes a few milliseconds, the evaluation process uses about 0.8 GB of RAM and the largest FAISS experiment uses about 1.8 GB according to /usr/bin/time -l. The e5-small model has about 117.7 million parameters (counted with sum(p.numel())), most of them in the multilingual vocabulary embedding table, so it runs well on a CPU. The model card also states that text longer than 512 tokens is truncated.

One environment bug worth recording, because you will hit it on macOS: loading sentence-transformers (PyTorch) and faiss in the same process made the program exit with code 139 (segmentation fault) on my first run, most likely because the two libraries load separate OpenMP runtimes. Setting OMP_NUM_THREADS=1 when running fixes it, and that is how I ran every script here. The KMP_DUPLICATE_LIB_OK=TRUE variable that the error message suggests is called unsafe by that same message, so I did not use it.

When not to use semantic search

  • The query is a code, a proper name or a serial number (order code HD-2024-000417, SKU): semantic search blurs exact characters because embeddings group similar-looking strings; BM25 or exact lookup does better. In practice you often combine the two (hybrid search) and merge the rankings.
  • You need to explain why a result came back: semantic search cannot point to the matching words, while keyword matching can show which words matched; vector distance cannot.
  • The corpus is small and the vocabulary is uniform: BM25 reaches R@1 0.58 versus 0.62 above, a difference of one question; it is not worth the added complexity if measuring on your own data gives the same picture.
  • The data is sensitive: treat the vector store like the source data for access control and retention, because vectors still carry information about the content.

Common mistakes

  1. Forgetting to normalize vectors in semantic search and then using the inner product, which makes ranking depend on vector length.
  2. Using an English model for Vietnamese semantic search: nomic-embed-text in this article loses even to BM25.
  3. Setting a fixed score threshold without calibrating it: E5 cosine scores are squeezed into a narrow band, and unrelated questions still score above 0.8.
  4. Using ANN too early for semantic search: at 200,000 vectors Flat is still fast; HNSW only adds 27 seconds of build time and 18% more memory.
  5. Evaluating semantic search with a few examples that look good instead of a question set with known answers, and forgetting to double-check the answer labels.
  6. Drawing conclusions from a gap of one or two questions when the test set has only 24.

Checklist for putting semantic search into production

  • Collect a few dozen to a few hundred real (anonymized) questions with correct answers; measure Recall@k and MRR, with BM25 as the baseline.
  • Choose the semantic search model by language and task type; check the model's required prefixes.
  • Normalize vectors and use IndexFlatIP until the data size and latency force a change.
  • Consider hybrid search (BM25 plus embeddings) and reranking a small top-k; measure the added latency.
  • Have a "no answer" path for semantic search and track failed questions so you can add data.
  • Save the semantic search index to disk and record the model version alongside the index; changing the model means re-embedding everything.

The semantic search code is in code/s7-35: demo_search.py, search_eval.py, models_rerank.py, faiss_index.py, plus the *_output.txt files and requirements.txt. The models download themselves from Hugging Face the first time; run the scripts with OMP_NUM_THREADS=1.

Next article

Choosing a vector database: FAISS, pgvector, Qdrant and HNSW indexes. FAISS is a library, not a database: the next article compares it with pgvector and Qdrant on metadata filtering, updates, replication, and how an HNSW index behaves when the data changes constantly. If you want to review how to phrase instructions to a model before combining them with search results, see Systematic prompt engineering: roles, examples and prompt testing.