In the previous article, TF-IDF treated "ô tô" and "xe hơi" (two Vietnamese words for "car") as two completely separate columns: nothing says they mean almost the same thing. The model only knows the two words show up in different news items. Word embeddings fix exactly this by representing each word as a real-valued vector of a few dozen to a few hundred dimensions, so that words that often appear in similar contexts sit close together in the space. This article trains Word2Vec (CBOW and Skip-gram) and FastText on 7.7 million tokens of Vietnamese news, looks at what the models learn and where they break, then measures whether word embeddings help the classifier from Vietnamese text classification: word segmentation, TF-IDF and linear models.

Environment: a 16 GB RAM machine running on CPU, Python 3.12, gensim 4.x, NumPy, scikit-learn and pyvi (exact versions are in requirements.txt). Word2Vec appeared in 2013 and FastText in 2016, so every word embedding technique here has been stable for a long time. The code is in code/s7-26. The vector, dot product and vector length concepts from NumPy for machine learning: vectorization and broadcasting are the foundation for the cosine section below.

The idea behind word embeddings: a word's meaning lives in the words around it

The distributional hypothesis says that words appearing in similar contexts have similar meanings. "Agribank" and "Vietcombank" both sit next to "lãi suất" (interest rate), "chi nhánh" (branch) and "tín dụng" (credit), so the pair ends up close together even though the model has no idea what a bank is. Instead of counting words, Word2Vec trains a shallow neural network (one hidden layer, no nonlinearity) on an auxiliary task: predict a word from its context, or predict the context from a word. We do not use the predictions themselves. We keep the weight matrix of the hidden layer, where each row is the vector of one word.

The original word embedding paper, Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013), proposed two lightweight architectures and reported training on 1.6 billion words in under a day. Two word embedding variants are widely used:

  • CBOW (continuous bag of words) averages the vectors of the surrounding words to predict the middle word. It is fast, smooths out noise, and suits frequent words.
  • Skip-gram takes one word and predicts each surrounding word. It is slower but usually gives better vectors for rare words.

The second paper, Distributed Representations of Words and Phrases and their Compositionality, added two tricks that made Word2Vec succeed in practice. The first is negative sampling, which replaces the full-vocabulary softmax with telling real pairs apart from a few noise pairs (the gensim default is negative=5). The second is subsampling of very frequent words such as "của" (of) and "và" (and), which speeds up training and makes the representation more even (default sample=0.001).

Preparing the corpus for word embeddings: segmentation and cleaning

Word embeddings learn from a very large amount of text, so for word embeddings the corpus matters more than the algorithm. We reuse the Vietnamese News Classification dataset (Apache-2.0) introduced in the previous article: up to 20,000 news items per category (title plus description), excluding exactly the 5,500 items that belong to the evaluation set so the classification step at the end is not contaminated, then sentence splitting, lowercasing and word segmentation with pyvi. Each sentence becomes one line in corpus_seg.txt, a format that gensim's LineSentence reads without loading the whole file into RAM. Real output:

220000 news items, 478656 sentences, 7691965 tokens, segmentation took 354s

Word segmentation is the most time-consuming step (354 seconds, including reading the parquet files), and it also leaves traces of error in the results, as we will see: pyvi sometimes joins words wrongly, producing tokens such as thi_đấu_đá or mu_xuất. Lowercasing makes "Mỹ" (America) and "mỹ" (from "mỹ phẩm", cosmetics) share one vector. For Vietnamese, lowercasing is usually still worth it because data is scarce, but you must know you are making that trade-off.

Training word embeddings: Word2Vec and FastText with gensim

The code uses one shared set of hyperparameters for a fair comparison: 100 dimensions, window=5, min_count=5 (drop words seen fewer than 5 times) and 5 epochs. It uses workers=1 and seed=1 so results are reproducible, because with multiple threads gensim's update order is nondeterministic. According to the gensim 4.x function signatures, the remaining defaults are negative=5, sample=0.001, alpha=0.025, and FastText uses character n-grams of length min_n=3 to max_n=6.

from gensim.models import Word2Vec, FastText
from gensim.models.word2vec import LineSentence

SENT = LineSentence("corpus_seg.txt")
COMMON = dict(vector_size=100, window=5, min_count=5, epochs=5, workers=1, seed=1)

cbow = Word2Vec(SENT, sg=0, **COMMON)
skip = Word2Vec(SENT, sg=1, **COMMON)
ft   = FastText(SENT, sg=1, min_n=3, max_n=6, **COMMON)

[training] Word2Vec CBOW                vocab= 27344     38s
[training] Word2Vec Skip-gram           vocab= 27344    100s
[training] FastText Skip-gram           vocab= 27344    301s

The cost of training word embeddings differs clearly: Skip-gram is about 2.6 times slower than CBOW, and FastText is 3 times slower than Skip-gram because each word must also update dozens of n-gram vectors. The cost also shows up in size: the Skip-gram model is 23 MB on disk, while FastText takes 823 MB, mostly the n-gram hash table (bucket=2000000 rows). Keep this in mind if you plan to deploy on a small machine.

Reading word embedding results: nearest neighbors

The simplest and most intuitive check for a Vietnamese word embedding model is to ask which words are closest by cosine. gensim's most_similar returns the words and their cosine scores. Real Skip-gram output:

ngân_hàng    agribank(0.77), tmcp(0.77), bidv(0.76), scb(0.76), vietinbank(0.75), nh(0.75)
bệnh_viện    bv(0.90), đa_khoa(0.83), phụ_sản(0.82), ung_bướu(0.78), medic(0.78), hồi_sức(0.77)
đại_học      đh(0.91), bách_khoa(0.82), cao_đẳng(0.82), sư_phạm(0.81), trường_thành_viên(0.78), trúng_tuyển(0.78)
ô_tô         ôtô(0.90), xe(0.88), xe_máy(0.88), gắn_máy(0.81), xe_tải(0.80), giường_nằm(0.80)
iphone       ipad(0.85), 5s(0.82), macbook(0.82), 6s(0.81), apple(0.81), galaxy(0.81)

The results show that word embeddings capture abbreviations ("bệnh viện" and "bv", "đại học" and "đh"), proper nouns of the same kind (banks, phone lines) and even spelling variants (ô_tô and ôtô, 0.90). Notice what they do not do: the "neighbors" are words with the same role in a sentence, not necessarily synonyms. "ô_tô" is close to "xe_máy" (motorbike) and "xe_tải" (truck) as one family of vehicles, and "giường_nằm" (sleeper bus) shows the news corpus leaking in. The pair "ô_tô" and "xe_hơi", which we would expect to be closest in meaning, reaches a cosine of only 0.674. The cause matches word frequency: counting with count_words.py on the corpus, "xe_hơi" appears only 250 times against 2,518 for "ô_tô", so its vector is learned from far fewer contexts.

Trying cosine on a few pairs shows that word embeddings distinguish degrees of relatedness: giáo_viên–học_sinh (teacher–student) scores 0.805, bác_sĩ–bệnh_nhân (doctor–patient) 0.727, while unrelated topics score very low (bóng_đá–ngân_hàng 0.276, bác_sĩ–cổ_phiếu 0.170). One pair in the test reported a missing word: tp.hcm is not in the vocabulary because pyvi splits it into tp, a period and hcm, and our punctuation filter drops the period. This is the first lesson: the string you type must go through exactly the same preprocessing as in training.

CBOW versus Skip-gram on the same corpus

CBOW:
ngân_hàng    doanh_nghiệp(0.65), scb(0.61), vietinbank(0.61), dn(0.61), bidv(0.59), kiểm_toán(0.58)
bóng_đá      giao_hữu(0.58), bóng_chuyền(0.56), u20(0.56), đội_tuyển(0.56), u19(0.55), trận_đấu(0.55)
Skip-gram:
bóng_đá      thi_đấu(0.72), sea_games(0.71), seagame(0.70), thi_đấu_đá(0.69), mu_xuất(0.67), mu_vs(0.66)

With CBOW word embeddings, the neighbors of "ngân_hàng" include "doanh_nghiệp" (enterprise) and "kiểm_toán" (auditing): words of the same topic rather than the same kind. Skip-gram returns mostly bank names. I do not draw strong conclusions from this small sample, and the cosine scores of the two models cannot be compared directly because their distributions have different scales. The window parameter also matters: with Skip-gram, a window of 2 gives all bank names (nh, scb, agribank), while a window of 10 starts adding same-topic words such as tín_dụng (0.79). Rule of thumb: a small window leans toward syntactic relations, a large window toward topic.

Vector analogies: impressive but fragile

What made word embeddings famous is the analogy such as "king − man + woman ≈ queen". In gensim that is most_similar(positive=[a, c], negative=[b]). The fastText tutorial shows the example berlin germany france returning "paris" with a score of 0.896, but that is a model trained on English Wikipedia, a hundred times larger than our corpus. Our results:

hà_nội - việt_nam + pháp  -> quận(0.58), nại(0.51), huyên(0.51)
paris - pháp + đức  -> tddc(0.56), phàn(0.55), phến(0.53)
tokyo - nhật_bản + hàn_quốc  -> triều_tiên(0.64), seoul(0.64), bangkok(0.62)
vua - nam + nữ  -> tài_ba(0.53), công_tử(0.53), lừng_lẫy(0.51)

Only one of the four analogies gives a half-reasonable result: "seoul" is in second place, after "triều_tiên" (North Korea). The other three return garbage, and in the first two, probably because "pháp" means both France and "law", so its vector is pulled in two directions (this is a hypothesis; I did not test it separately). Do not put word embedding analogies in a customer demo before you have tried them on your own data: they are only reliable when the corpus is large enough and the words are frequent enough. Seven million tokens is small for word embeddings compared with published models, and that number should make us more humble when we read the pretty 2D plots found online.

FastText: handling unseen words with character n-grams

Word2Vec-style word embeddings give each word its own vector. A word that is not in the vocabulary immediately raises KeyError, even when it is only a small variant of a known word. Enriching Word Vectors with Subword Information (Bojanowski et al., 2016) proposed FastText: each word is the sum of the vectors of its character n-grams, so a new word still gets a vector. Trying four words that are not in the vocabulary:

ngân_hàng_số   Skip-gram: KeyError  | FastText: ngân_hàng(0.93), ngân_hàng_tnhh(0.88), ngân_hàng_đầu_tư(0.84), ngân_hà(0.82)
bóng_đá_nữ     Skip-gram: KeyError  | FastText: bóng_đá(0.92), bóng_đè(0.89), bóng_chày(0.78), bóng_bàn(0.76)
xe_điện_hóa    Skip-gram: KeyError  | FastText: xe_tải_điện(0.79), điện_phân(0.76), cơ_điện_tử(0.76), điện_áp(0.75)
covid19        Skip-gram: KeyError  | FastText: covid(0.98), 19(0.91), covivac(0.82), covax(0.76)

FastText word embeddings correctly place "ngân_hàng_số" (digital bank) near "ngân_hàng" (0.93) and "covid19" near "covid" (0.98). But there is a downside: "bóng_đè" (an unrelated word that merely looks similar) ranks second for "bóng_đá_nữ" (women's football) at 0.89, and "ngân_hà" (galaxy) makes the top list for "ngân_hàng_số". Because the vectors rely on spelling, words that look alike but mean different things get pulled together. In Vietnamese, where compounds are joined with underscores and many different words differ only by a diacritic, this is not rare. FastText is therefore most useful for spelling errors, new compounds and morphologically rich languages, and it needs manual checking before you trust it.

Word embeddings as classification features: a real measurement

The practical question for word embeddings: if we replace TF-IDF with word embeddings, does text classification improve? The simplest approach is to average the vectors of the words in an item and feed the result to a linear model. We use exactly the 5,500-item set from the previous article with the same 80/20 split and random_state=42; the word embeddings did not see any test item while learning:

def mean_vec(model, tokens_list, use_oov=False):
    out = []
    for toks in tokens_list:
        vs = [model.wv[w] for w in toks if use_oov or w in model.wv.key_to_index]
        out.append(np.mean(vs, axis=0) if vs else np.zeros(model.vector_size))
    return np.array(out)

mean Word2Vec CBOW: out-of-vocabulary token rate = 1.34%
  + LinearSVC             acc=0.7409 macroF1=0.7378
  + LogisticRegression    acc=0.7264 macroF1=0.7260
mean Word2Vec Skip-gram: out-of-vocabulary token rate = 1.34%
  + LinearSVC             acc=0.7545 macroF1=0.7506
  + LogisticRegression    acc=0.7455 macroF1=0.7436
mean FastText (with OOV): out-of-vocabulary token rate = 0.00%
  + LinearSVC             acc=0.7609 macroF1=0.7572
  + LogisticRegression    acc=0.7509 macroF1=0.7496
TF-IDF 1-2 gram + LinearSVC (same data split)  acc=0.7527 macroF1=0.7513

The word embedding results need careful reading. Averaged Skip-gram (macro-F1 0.7506) and FastText (0.7572) are on par with TF-IDF (0.7513), not clearly ahead. In the previous article, the standard deviation across cross-validation folds was about 0.015, larger than every difference here, so we cannot say word embeddings "win". Only two observations are solid enough: CBOW is about 1.3 points worse than Skip-gram in this experiment, and averaging vectors, which throws away word order, still matches the quality of a TF-IDF pipeline with bigrams.

So where are word embeddings useful if they do not beat TF-IDF? When you have very few labeled examples but plenty of unlabeled text, you train word embeddings on all the raw text and then fit the classifier on a few hundred samples. In this experiment we have 4,400 labeled samples, so TF-IDF has not lost yet. A 100-dimensional word embedding vector is also dense and compact: one news item is 100 numbers, versus a sparse TF-IDF vector of more than 20 thousand dimensions, which is convenient for approximate search and clustering. I did not run the low-label experiment in this article, so I have no numbers to conclude with; it is a direction worth trying.

When not to use static word embeddings

  • One word, one vector: with static word embeddings, "bank" as a riverbank and "bank" as a financial institution, or "pháp" above, share a single vector. Polysemy is an inherent limit; contextual models such as Transformers, the topic of the next two articles, produce a different vector depending on the sentence.
  • Small or off-domain corpus: word embeddings only reflect their corpus; trained on news, "giường_nằm" becomes a neighbor of "ô_tô". For specialized domains such as medicine or law, you need an in-domain corpus.
  • Negation and word order: averaging vectors cannot tell "good, not bad" from "bad, not good".
  • Bias in the data: word embeddings also learn the prejudices of their corpus; when you use them to rank résumés or score people, you must test for bias.

Common mistakes with word embeddings

  1. Preprocessing at use time differs from training time: lowercasing, word segmentation and punctuation removal must be identical, otherwise words fall outside the vocabulary (like tp.hcm above).
  2. Judging by a few nice examples: picking the successful examples and concluding the model is good. List the failed ones too, like the four analogies above.
  3. Running many threads and expecting identical results: with workers greater than 1, two runs give different vectors. Record this if you need reproducibility, and run a single thread as this article does.
  4. Forgetting min_count: a word seen once or twice gets an almost random vector; raise min_count or use FastText.
  5. Loading the heavy FastText model even when you do not need it: the n-gram table takes 823 MB. If you do not need unseen words, use the 23 MB Word2Vec model.
  6. Ignoring warnings in the log: in this run, gensim 4.x on Python 3.12 printed 129 lines (not counting merged duplicates, so the real number is higher) of Exception ignored in: 'gensim.models.word2vec_inner.our_dot_float' to stderr (saved in stderr.txt). The check_dot.py script retrains a small model (the first 20,000 sentences, 2 epochs) and confirms that every vector is finite and that the cosine returned by gensim matches a hand calculation in NumPy (pair ngân_hàng and bidv: 0.8810 versus 0.8810; output in check_dot_output.txt, and this run had no warning lines), but I did not find the root cause; if you go to production, pin the versions and run a similar check yourself.

Word embedding checklist

  • For every word embedding project, keep word segmentation and normalization consistent between training and use; store the preprocessing code too.
  • Start with Skip-gram for a medium corpus, CBOW when you need speed, and FastText when there are many unseen words or spelling errors.
  • Set min_count, window and vector_size deliberately; record seed and workers=1 if you need reproducibility.
  • Check word embeddings with nearest neighbors and cosine on word pairs from your own domain, including the failing pairs.
  • Compare against a TF-IDF baseline with the same split, and remember the run-to-run error.
  • Weigh the model size (23 MB versus 823 MB) before deploying.

The word embedding source code consists of prepare_corpus.py, embed_demo.py, downstream.py, plus count_words.py, check_dot.py with output.txt, count_output.txt, check_dot_output.txt, downstream_output.txt, stderr.txt and requirements.txt. The model files (*.model, *.npy, almost 900 MB in total) and corpus_seg.txt are not included; rerun prepare_corpus.py and then embed_demo.py to recreate them. All word embedding figures are real output on a 16 GB machine, single thread, 5 epochs; the corpus is public news and contains no customer personal data.

Next article

Transformer and attention: understanding the architecture through code. Static word embeddings assign one vector to each word regardless of context. The next article builds the attention mechanism in code so that each word gets a vector that depends on the whole sentence, the foundation of PhoBERT and large language models.