Fine-tuning PhoBERT with Hugging Face for Vietnamese sentiment analysis

A store gets thousands of customer reviews every month, and the usual question is how many of them are complaints. This task is called sentiment analysis, and for Vietnamese the popular choice today is PhoBERT, a language model pre-trained specifically for Vietnamese. This article goes through the full loop: download the UIT-VSFC student feedback dataset, build a TF-IDF baseline as the bar to beat, fine-tune PhoBERT with the Hugging Face Trainer on an Apple Silicon machine, try class weights for the rare label, and then read the real results through a confusion matrix instead of looking at accuracy alone. Every number below is real output from a run on this machine.

This article follows Transformer and attention: understanding the architecture through code and reuses the baseline from Vietnamese text classification: word segmentation, TF-IDF and linear models. Versions used: Python 3.12, torch 2.2.2, transformers 4.38.2, datasets 2.19.0, scikit-learn 1.4.2, pyvi 0.1.1. The evaluation_strategy argument in the code is correct for transformers 4.38; if you install a different version, check the documentation for that version.

What PhoBERT is and why not multilingual BERT

According to the paper PhoBERT: Pre-trained language models for Vietnamese by Dat Quoc Nguyen and Anh Tuan Nguyen (Findings of EMNLP 2020), PhoBERT comes in base and large versions and was the first public large-scale monolingual language model for Vietnamese. The authors report that it outperforms XLM-R, a strong multilingual model, and reached state-of-the-art results at the time on part-of-speech tagging, dependency parsing, named-entity recognition and natural language inference. The architecture is the same as BERT (a Transformer encoder, covered in the previous article); the differences are the pre-training data and the tokenizer.

The table in the VinAIResearch/PhoBERT repository lists: phobert-base has 135 million parameters, phobert-large has 370 million, the maximum length is 256 tokens, pre-training used 20 GB of Wikipedia and news text, and the license is MIT. This article uses phobert-base because it fits the RAM limit of the test machine and is enough to demonstrate the PhoBERT fine-tuning workflow.

A note on versions: the same repository also has phobert-base-v2, pre-trained on an additional 120 GB of text from OSCAR-2301 and licensed under AGPL-3.0 (the Hugging Face model page shows a 2023 copyright; the repository page gives no release date). This article uses v1 (vinai/phobert-base, MIT); if you use v2 or another version, read the license of that exact version again.

A hard requirement: word-segment the text first

This is where beginners break things most often. The PhoBERT README states that the input must be word-segmented beforehand, and recommends the RDRSegmenter from VnCoreNLP because the pre-training data was segmented with that tool. Vietnamese is written with spaces between syllables, not between words: "giảng viên" (lecturer) is one word made of two syllables. After segmentation, the syllables of a compound word are joined with underscores. In this article I use pyvi because it is lightweight (no Java needed); it is a different tool from RDRSegmenter, so word boundaries may differ slightly from those seen during pre-training, which is a point that could lower quality compared with the standard approach.

The tokenize_demo.py script shows how the tokenizer handles the two cases (the example is a Vietnamese sentence meaning "the lecturer is enthusiastic and teaches in an easy-to-understand way"):

original  : giảng viên nhiệt tình và giảng dạy dễ hiểu .
segmented : giảng_viên nhiệt_tình và giảng_dạy dễ hiểu .
tokens (segmented)  : ['giảng_viên', 'nhiệt_tình', 'và', 'giảng_dạy', 'dễ', 'hiểu', '.']
tokens (not segmented): ['giảng', 'viên', 'nhiệt', 'tình', 'và', 'giảng', 'dạy', 'dễ', 'hiểu', '.']

With segmentation, "giảng_viên" is in the vocabulary as a single token. Without it, PhoBERT receives loose syllables, which differ from what it saw during pre-training. I did not measure the effect on the score separately in this article, so I do not claim a specific drop; you should test it yourself if your data is hard to segment.

The UIT-VSFC dataset and the TF-IDF baseline

The dataset is UIT-VSFC (Vietnamese Students' Feedback Corpus) from the UIT NLP group, published by Nguyen et al. at the KSE 2018 conference. The dataset card on Hugging Face lists 11,426 training sentences, 1,583 validation sentences and 3,166 test sentences, each with a sentiment label (0 negative, 1 neutral, 2 positive) and a topic label. The license is listed as "Unknown", so for a commercial project check the terms before using it; this article uses it for learning purposes only. Real statistics after downloading (prepare_output.txt):

train       11426 sentences | negative 5325 (46.6%) | neutral 458 (4.0%) | positive 5643 (49.4%)
            mean length 14.3 syllables, longest 159
validation   1583 sentences | negative 705 (44.5%) | neutral 73 (4.6%) | positive 805 (50.9%)
test         3166 sentences | negative 1409 (44.5%) | neutral 167 (5.3%) | positive 1590 (50.2%)
raw example      : nói tiếng anh lưu loát .
after segmenting : nói tiếng anh lưu_loát .

The point to notice right away: the "neutral" label makes up only about 4–5%. This is imbalanced data, so overall accuracy will hide the quality on the rare class; we met this problem in Evaluating classification models: precision, recall, ROC and skewed data. That is why this article reports macro-F1 (the average F1 of the three classes, each with equal weight) next to accuracy.

The baseline: TF-IDF with 1–2 grams and LinearSVC on the segmented text, with the C parameter chosen on validation (baseline.py):

C=0.1: validation macro-F1 = 0.6630
C=0.3: validation macro-F1 = 0.7237
C=1.0: validation macro-F1 = 0.7464

Chose C=1.0 (by validation). Test results (3166 sentences):
accuracy = 0.8986 | macro-F1 = 0.7017
              precision    recall  f1-score   support
    negative      0.881     0.957     0.917      1409
     neutral      0.622     0.168     0.264       167
    positive      0.924     0.923     0.924      1590

The baseline reaches 89.9% accuracy but catches only 16.8% of neutral sentences (recall 0.168). The bar to beat is accuracy 0.8986 and macro-F1 0.7017.

Fine-tuning PhoBERT with the Hugging Face Trainer

The Hugging Face sequence classification guide describes the standard workflow: load AutoModelForSequenceClassification with the number of labels, tokenize the text, use DataCollatorWithPadding to pad dynamically per batch instead of padding the whole set to the maximum length, define compute_metrics, and pass everything to a Trainer. With PhoBERT, AutoModelForSequenceClassification returns RobertaForSequenceClassification: the classification head (dropout, a linear layer, tanh, dropout, and an output linear layer) sits on the vector of the leading <s> token and is randomly initialized. All parameters, about 135 million including the new head, are updated together, which is why this is called fine-tuning.

The data and model loading part of train.py:

tok = AutoTokenizer.from_pretrained("vinai/phobert-base")
class DS(torch.utils.data.Dataset):
    def __init__(self, rows):
        enc = tok([r["seg"] for r in rows], truncation=True, max_length=64)
        self.items = [{"input_ids": enc["input_ids"][i], "attention_mask": enc["attention_mask"][i],
                       "labels": r["label"]} for i, r in enumerate(rows)]
    def __len__(self): return len(self.items)
    def __getitem__(self, i): return self.items[i]

model = AutoModelForSequenceClassification.from_pretrained(
    "vinai/phobert-base", num_labels=3,
    id2label=dict(enumerate(LABELS)), label2id={l: i for i, l in enumerate(LABELS)})

Choosing max_length=64 instead of 256 is justified: the real output shows a mean token length of 13.5, a 99th percentile of 46, and only 0.24% of training sentences truncated at 64. With sentences this short, keeping 256 only costs extra memory. If your data is multi-paragraph reviews, measure the length distribution again before settling on a value.

Hyperparameters and training:

args = TrainingArguments(
    output_dir="tmp_out", num_train_epochs=3, learning_rate=3e-5, weight_decay=0.01, warmup_ratio=0.1,
    per_device_train_batch_size=16, per_device_eval_batch_size=64, evaluation_strategy="epoch",
    save_strategy="no", logging_steps=100, report_to=[], seed=42, dataloader_num_workers=0)
trainer = Trainer(model=model, args=args, train_dataset=dtr, eval_dataset=dva, tokenizer=tok,
                  data_collator=DataCollatorWithPadding(tok), compute_metrics=metrics)
trainer.train()

A few choices deserve an explanation. A learning rate of 3e-5 is within the range commonly used for fine-tuning BERT-sized models (roughly 2e-5 to 5e-5). warmup_ratio=0.1 raises the learning rate gradually over the first 10% of steps and then decays it linearly to 0; this helps avoid damaging the pre-trained weights while the new classification head is still random. save_strategy="no" writes no checkpoints, so it costs no disk, but in return we have no "best model by validation" and use the model after the last epoch. I also set torch.set_num_threads(2) to respect the resource limits of a shared machine.

The result of the full run, 3 epochs on 11,426 sentences (train_plain_output.txt), on the Apple Silicon MPS device:

training time: 10.2 minutes
epoch 1: val loss 0.2568 | acc 0.9406 | macro-F1 0.8329
epoch 2: val loss 0.2032 | acc 0.9551 | macro-F1 0.8606
epoch 3: val loss 0.2217 | acc 0.9558 | macro-F1 0.8672

TEST (3166 sentences): accuracy = 0.9397 | macro-F1 = 0.8253
              precision    recall  f1-score   support
    negative      0.941     0.972     0.956      1409
     neutral      0.738     0.455     0.563       167
    positive      0.951     0.962     0.957      1590

confusion matrix (row = true, column = predicted):
 [[1369   16   24]
 [  37   76   54]
 [  49   11 1530]]
peak RAM (ru_maxrss): 2.20 GB

Before reading the PhoBERT results, I need to mention the quick smoke test I ran earlier with 320 sentences and 1 epoch: accuracy 0.7716, macro-F1 0.5280, and the "neutral" class was ignored completely (precision and recall both 0). Running a small set first before a long run is a good habit for catching code bugs, but do not read its score as model quality.

PhoBERT results compared with the baseline

PhoBERT beats the baseline by 4.1 percentage points of accuracy (0.8986 to 0.9397) and 12.4 points of macro-F1 (0.7017 to 0.8253). Most of PhoBERT's advantage comes from the rare class: the F1 of "neutral" rises from 0.264 to 0.563, and recall from 0.168 to 0.455. The two large classes improve too, but less (negative F1 from 0.917 to 0.956, positive from 0.924 to 0.957).

Reading the confusion matrix shows where it is still weak. Of the 167 truly neutral sentences, PhoBERT gets only 76 right; 54 are labeled "positive" and 37 are labeled "negative". Some examples of neutral sentences in the test set (translated from Vietnamese): "it's fine." and "this course helps us understand the basic issues." These are sentences where the human annotators themselves may disagree (this article does not measure inter-annotator agreement, so that is a guess, not a conclusion), and if so, the quality ceiling for this class is lower than for the other two, and a larger model may not fix it. If your use case needs to catch neutral opinions correctly, check label quality before increasing model size.

Imbalanced data: class weights for PhoBERT

The common approach for a rare class is to multiply the cross-entropy loss by weights inversely proportional to class frequency. The Trainer documentation shows how to override compute_loss for this case. In train.py, the --weighted flag turns on the following subclass:

class WTrainer(Trainer):
    def compute_loss(self, model, inputs, return_outputs=False):
        labels = inputs.pop("labels"); out = model(**inputs)
        loss = nn.functional.cross_entropy(out.logits, labels, weight=self.w.to(out.logits.device))
        return (loss, out) if return_outputs else loss

cnt = np.bincount([r["label"] for r in tr], minlength=3)
trainer.w = torch.tensor(len(tr) / (3 * cnt), dtype=torch.float32)

With training frequencies of 5,325 / 458 / 5,643, this formula gives the weights printed during the real run: [0.72, 8.32, 0.67], meaning one neutral sentence "weighs" about 12 times as much as one positive sentence in the loss. The result after 3 epochs (train_weighted_output.txt):

epoch 1: val loss 0.8277 | acc 0.9463 | macro-F1 0.8522
epoch 2: val loss 0.6503 | acc 0.9526 | macro-F1 0.8672
epoch 3: val loss 0.7835 | acc 0.9551 | macro-F1 0.8718

TEST (3166 sentences): accuracy = 0.9378 | macro-F1 = 0.8291
    negative      0.941     0.967     0.954      1409
     neutral      0.717     0.485     0.579       167
    positive      0.950     0.959     0.955      1590

confusion matrix (row = true, column = predicted):
 [[1363   17   29]
 [  35   81   51]
 [  50   15 1525]]

This needs a careful reading. Macro-F1 rises from 0.8253 to 0.8291 (about 0.4 points) and neutral recall from 0.455 to 0.485, which is 5 more sentences caught out of 167; in return, accuracy drops by 0.19 points. That is a small difference, from one random seed and one run per configuration; I have no confidence interval, so I cannot say class weights are "better" with certainty. What can be said: on this data, class weights shift the trade-off slightly toward recall on the rare class, and they are not magic. For a firmer conclusion, you need to run several seeds and compare the mean with the standard deviation.

Also note that the weighted run's validation loss (0.78) being higher than the plain run's (0.22) does not mean the model is worse: the two losses multiply by different weights, so they cannot be compared directly.

Resources: running PhoBERT on 16 GB of RAM

Both fine-tuning runs peaked at about 2.2 GB of RAM (ru_maxrss 2.20 and 2.21 GB), and each took about 10 minutes for 3 epochs on MPS. The measures used: batch size 16, maximum length 64 (dynamic padding, not padding to 256), 2 CPU threads, and no checkpoints. The RAM figure in this article is ru_maxrss, the peak resident set size of the process as recorded by the operating system. If you hit an out-of-memory error on MPS, reduce the batch size, reduce max_length, or call torch.mps.empty_cache() periodically, because dynamic padding creates many different tensor shapes and the cache grows over time.

The test machine has no NVIDIA GPU, so I did not run the fp16 part in this article.

Using PhoBERT after training

With a trained PhoBERT, the last step is packaging it for inference. After saving with model.save_pretrained() and tok.save_pretrained(), you reload it with pipeline("text-classification", ...). Remember that the input must go through the same word segmentation as during training, otherwise results drift silently without any error:

from pyvi import ViTokenizer
from transformers import pipeline

clf = pipeline("text-classification", model="model_out", tokenizer="model_out")
text = "giảng viên nhiệt tình nhưng phòng học quá nóng ."
print(clf(ViTokenizer.tokenize(text)))

In this article I only set the SAVE_MODEL=1 flag to save the model when needed and do not save it by default, because a fine-tuned PhoBERT is larger than 500 MB. The inference snippet above is an API sample; I did not run it separately and do not paste any output.

When not to fine-tune PhoBERT

• Little data (a few hundred sentences) or a simple task: the TF-IDF baseline is often enough, cheaper and easier to explain, as in the text classification article. Here the baseline already reached 89.9% accuracy; ask yourself whether 4 extra points are worth the operating cost.
• Long text: PhoBERT is limited to 256 tokens; for long documents you need to split into chunks or choose another model.
• A domain far from news and Wikipedia: social media with many abbreviations, teencode and slang can make segmentation and the vocabulary perform poorly. Measure on your own real data.
• You need to control licensing: phobert-base and phobert-large use the MIT license according to the table in the VinAIResearch repository (phobert-base-v2 uses AGPL-3.0, which is very different), and the UIT-VSFC dataset says "Unknown"; read carefully before commercial use.
• You need to explain each decision: deep models are harder to explain than linear ones; see Explaining models with SHAP: understanding why a model decides if you need to add that.

Common mistakes

1. Forgetting word segmentation: PhoBERT still runs but receives loose syllables, different from what it learned.
2. Looking only at accuracy: with skewed data, 94% accuracy can go together with 45% recall on the rare class, as in the table above.
3. Choosing a model by test score: parameters and epochs should be chosen on validation, and the test set touched only once at the end; in this article the baseline's C was also chosen on validation.
4. Drawing conclusions from one seed: a 0.4-point macro-F1 gap between two configurations is not enough evidence.
5. API version drift: argument names like evaluation_strategy (version 4.38 is used here) may change in later transformers versions; pin the version in requirements.txt.
6. Letting checkpoints pile up: each checkpoint is hundreds of MB; set save_total_limit or turn saving off as in this article.

PhoBERT fine-tuning checklist

• Look at the label distribution first; choose macro-F1 when the data is skewed.
• Build a TF-IDF baseline first to know whether a deep model is worth it.
• Segment the input the same way in training and inference.
• Measure the token length distribution to choose max_length; use dynamic padding.
• Run 1 epoch on a small set to catch code bugs.
• Compare configurations over several seeds before claiming an improvement.
• Read the confusion matrix and the wrong sentences; check label quality.
• Check the licenses of the data and the model.

The code lives in code/s7-28: prepare.py, baseline.py, train.py, tokenize_demo.py and the *_output.txt files. The data is downloaded from the Hugging Face Hub (uitnlp/vietnamese_students_feedback) and saved as parquet and JSONL in data/. The numbers in this article are real output from one run with seed=42 on Apple Silicon; on another machine results will differ slightly.

Next article

Vietnamese speech-to-text with Whisper: running locally and speeding it up. After working with text, the next article moves to audio: running the Whisper model on CPU, measuring WER for Vietnamese and comparing speed-up options such as int8 quantization and silence filtering.