Vietnamese speech-to-text with Whisper: running locally and speeding it up

Turning speech into text is a familiar problem for businesses: call-center recordings, meeting minutes, subtitles for tutorial videos. Since late 2022, OpenAI's Whisper has made it something you can run on a laptop without sending recordings to a third party. This article runs Whisper on the CPU of an Apple Silicon machine, measures accuracy (WER) and speed (RTF) on Vietnamese, compares openai-whisper with faster-whisper, and then measures each optimization option one by one: int8 quantization, beam size, and silence filtering with VAD. It ends with experiments on noise and on the model "making up" text when it hits silence, which is exactly what causes incidents in real products.

The article builds on Transformer and attention: understanding the architecture through code (Whisper is a Transformer encoder–decoder) and on the measurement methods from Model evaluation for classifiers: precision, recall, ROC and imbalanced data. Environment: Python 3.12, openai-whisper 20231117, faster-whisper 1.0.2, ctranslate2 4.2.1, torch 2.3.0, CPU only, with the base and small models. Every number is real output from this machine. Timings will differ on yours, but the ratios between configurations are much more stable. The test sentences, references and transcripts are in Vietnamese because that is the language being recognized; an English gloss follows wherever it matters.

What Whisper is and why run it locally

According to the paper Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., 2022), Whisper was trained to predict transcripts on 680,000 hours of audio collected from the internet, multilingual and multitask, without dataset-specific fine-tuning. The model card on Hugging Face spells out the data mix: about 65% (438,000 hours) is English, 18% (126,000 hours) is non-English audio with English transcripts, and 17% (117,000 hours) is non-English audio with transcripts in the same language, spread over 98 languages. That means Vietnamese gets only a small slice of those 117,000 hours, and quality will trail English. The measurements below put concrete numbers on that gap.

The architecture is a Transformer encoder–decoder: audio is cut into 30-second windows and converted to a log-Mel spectrogram, the encoder encodes it, and the decoder generates text tokens one by one. The documentation of the openai/whisper repository explains that transcribe() reads the whole file and processes it with a sliding 30-second window, running autoregressive prediction on each window. The multilingual model sizes from that repository's table:

Speeds are relative to large, measured on an A100. This article runs only base and small because of the RAM limit of the test machine and because the goal is to learn how to measure; medium and above are not run here.

The reason to run locally: recordings contain customer information that should not leave your infrastructure, and the cost is not billed per minute. The price is that you measure quality and tune speed yourself.

Preparing test data with ground truth

To compute WER you need a correct transcript. To avoid using real voices or personal data, I wrote six customer-service sentences in Vietnamese and read them with the built-in macOS synthetic voice (say -v Linh), then converted them to 16 kHz mono 16-bit WAV (make_audio.py):

SENTS = [
    "Xin chào quý khách, đơn hàng của anh đã được xác nhận và sẽ giao trong hai ngày tới.",   # "Hello, your order is confirmed and will be delivered within two days."
    "Cửa hàng mở cửa từ tám giờ sáng đến chín giờ tối, kể cả thứ bảy và chủ nhật.",           # "The store is open from 8 a.m. to 9 p.m., including Saturday and Sunday."
    # ... four more sentences about payment, warranty, confirmation messages, technical support
]
for i, s in enumerate(SENTS, 1):
    subprocess.run(["say", "-v", "Linh", "-o", f"audio/s{i}.aiff", s], check=True)
    subprocess.run(["afconvert", "-f", "WAVE", "-d", "LEI16@16000", "-c", "1",
                    f"audio/s{i}.aiff", f"audio/s{i}.wav"], check=True)

The total is 34.8 seconds of audio. The limitation needs to be stated plainly: the synthetic voice is clean and even, with no background noise and no regional accent. It lets you compare configurations on identical input, but the absolute WER here does not represent real call-center recordings. For a real project, collect a few dozen minutes of your own recordings (with permission) and transcribe them by hand as the reference.

The normalization function (lowercase, NFC, strip punctuation) also affects WER, because Whisper adds capitalization and punctuation on its own.

def norm(s):
    s = unicodedata.normalize("NFC", s.lower())
    s = re.sub(r"[^\w\s]", " ", s)
    return re.sub(r"\s+", " ", s).strip()

wer = jiwer.wer([norm(v) for v in refs.values()], [norm(h) for h in hyps])

Running Whisper with openai-whisper and faster-whisper

Both approaches run the same set of weights. openai-whisper is the PyTorch reference implementation. faster-whisper reimplements the model on CTranslate2, an inference engine for Transformers. The project README claims up to 4 times faster at the same accuracy, with less memory, and says 8-bit quantization improves efficiency further on both CPU and GPU. Two minimal calls (transcribe_compare.py):

# openai-whisper (PyTorch, fp32, CPU)
import whisper
m = whisper.load_model("small", device="cpu")
text = m.transcribe(audio, language="vi", fp16=False, beam_size=5, temperature=0.0)["text"]

# faster-whisper (CTranslate2, int8, CPU)
from faster_whisper import WhisperModel
m = WhisperModel("small", device="cpu", compute_type="int8")
segs, info = m.transcribe(audio, language="vi", beam_size=5, temperature=0.0)
text = " ".join(s.text for s in segs)

Note that language="vi" skips the automatic language detection step. Also, transcribe() in faster-whisper returns a generator: decoding only runs when you iterate over segs, so if you time the call without consuming the generator you get a deceptively good number. In the benchmark code, I join the text inside the timed section.

Results on 34.8 seconds of audio with the small model (out_small.txt):

model=small  files=6  total audio=34.8s
[openai-whisper fp32 CPU] load=1.9s  inference=33.8s  RTF=0.971  WER=9.6%  CER=6.5%
[faster-whisper int8 CPU] load=2.4s  inference=15.8s  RTF=0.455  WER=9.6%  CER=6.5%

With the base model (out_base.txt):

model=base  files=6  total audio=34.8s
[openai-whisper fp32 CPU] load=1.2s  inference=14.4s  RTF=0.413  WER=16.5%  CER=8.1%
[faster-whisper int8 CPU] load=2.1s  inference=7.0s  RTF=0.200  WER=14.8%  CER=7.3%

RTF (real-time factor) is processing time divided by audio duration; below 1 means faster than real time. In short:

• Model small: faster-whisper int8 is about 2.1 times faster (RTF 0.455 versus 0.971) with an identical WER of 9.6%.
• Model base: about 2.1 times faster (RTF 0.200 versus 0.413); WER is even slightly lower (14.8% versus 16.5%) thanks to small differences in decoding, so do not read this as "int8 is more accurate".
• From base to small, WER drops from 14.8% to 9.6% but the time more than doubles (about 2.3 times: 7.0 seconds to 15.8 seconds with faster-whisper, 14.4 seconds to 33.8 seconds with openai-whisper). This is the typical model-size trade-off.

The 2.1x figure is smaller than the README's "up to 4 times". There is no contradiction: the README measured 13 minutes of audio on an Intel i7-12700K with 8 threads and beam 5, and its table lists openai/whisper fp32 at 6 minutes 58 seconds, faster-whisper fp32 at 2 minutes 37 seconds, and faster-whisper int8 at 1 minute 42 seconds. The ratio between original fp32 and int8 is about 4.1x on that configuration. On the M4, PyTorch is already quite good on Arm CPUs, so the gap narrows. The lesson is that you must measure on the exact hardware you will deploy to.

Where it goes wrong: reading sentence by sentence

WER is just one number; you only learn whether the system is usable by reading the individual errors. With faster-whisper small, sentences s4 and s5 gave:

s4.wav REF: Sản phẩm này được bảo hành mười hai tháng, đổi trả miễn phí trong vòng bảy ngày.
       HYP: Sản phẩm này được bảo hành 12 tháng, đổi trả miễn phí trong vòng bài ngày.
s5.wav REF: Hệ thống sẽ gửi tin nhắn xác nhận đến số điện thoại đã đăng ký của quý khách.
       HYP: Hệ thống sẽ gửi tin nhánh xác nhận đến số điện thoại đã đăng ký của quý khách.

Three kinds of errors repeat. Numbers written as words are converted to digits ("mười hai" becomes "12", "tám giờ" becomes "8h"): this is a difference in writing convention that inflates WER artificially, so normalize both sides if you want a pure measurement. Wrong tone marks or similar-sounding words ("bảy" (seven) becomes "bài" (lesson), "tin nhắn" (message) becomes "tin nhánh" (a non-word)): these are real errors for Vietnamese, where the tone mark decides the meaning; reduce them with a larger model or a domain vocabulary. Wrong initial consonants ("Khách" (guest) becomes "Khác" (other)): the sentence still reads fluently, so the error only shows up when you compare against a hand-made transcript. "Bài ngày" instead of "bảy ngày" (seven days) corrupts the warranty information, so important numbers need human confirmation.

Speeding up Whisper: int8, beam size and VAD

All three options live directly in the faster-whisper call. A single measurement is not reliable: my first run gave the opposite result from the rerun, because the first configuration paid for a cold start and the load of a shared machine. So the final script (bench_options.py) preloads both models, runs one discarded warm-up pass, then measures each configuration 3 times in rotating order so that no configuration always runs first, and reports the median along with the min–max range. The data is the six sentences stitched into 158.3 seconds of audio (18 sentences: three loops, with 3 seconds of silence between sentences), model small, condition_on_previous_text=False:

audio length 158.3s (18 sentences, with 3s silences)
median of 3 runs (min-max); inference time, RTF = time / audio length
int8     beam=5 vad=False   17.1s (16.1-18.9)  RTF=0.108  WER=9.9%
int8     beam=1 vad=False   10.4s (10.4-10.7)  RTF=0.066  WER=9.6%
int8     beam=1 vad=True    10.3s (9.9-11.4)  RTF=0.065  WER=9.3%
float32  beam=5 vad=False   20.9s (20.1-21.1)  RTF=0.132  WER=9.6%
float32  beam=1 vad=False   10.3s (10.1-10.7)  RTF=0.065  WER=9.9%
float32  beam=1 vad=True    10.1s (9.8-10.7)  RTF=0.064  WER=9.6%

This benchmark process peaks at about 2.25 GB RSS according to /usr/bin/time -l (holding two small models at once, int8 and float32). A few things to read from the table, with a caveat about sample size (a single synthetic voice, so differences under 0.5 WER points are within measurement noise):

• Beam size is the clearest and most repeatable lever: dropping the beam from 5 to 1 speeds int8 up from 17.1 seconds to 10.4 seconds and float32 from 20.9 seconds to 10.3 seconds, a gap far larger than the spread between runs, while WER does not get worse on this clean data. With clean speech, greedy decoding is enough; with noisy audio, a larger beam may be worth it.
• int8 quantization makes no significant speed difference at beam=1 on the M4: 10.4 seconds versus 10.3 seconds, while the runs of each configuration already varied by 0.3 to 1.5 seconds. At beam=5, int8 is faster than float32 (17.1 seconds versus 20.9 seconds), but that is only one pair of numbers, so I do not generalize it. The CTranslate2 quantization documentation explains that int8 quantizes the weights of embedding and linear layers while the other layers run at their original precision, and the main benefit is memory (for example, a Transformer base model shrinks from 364 MB to about 100 MB), while speed depends on hardware. The safe conclusion: choose int8 to save RAM, do not expect it to be faster by itself, and measure again on your hardware.
• VAD (vad_filter=True) is not a speedup tool: the timing differences (10.4 seconds to 10.3 seconds for int8, 10.3 seconds to 10.1 seconds for float32) are within noise, and my very first single measurement showed a much larger "saving", probably due to run order and machine load. The reason to turn VAD on is to stop the model from making up text over silence, as the experiment below shows.

The faster-whisper README describes vad_filter as using the Silero VAD model to drop segments without speech, and by default it removes only silences longer than 2 seconds; you can tune it with vad_parameters, for example min_silence_duration_ms=500. A suggested configuration:

model = WhisperModel("small", device="cpu", compute_type="int8")
segs, info = model.transcribe(
    "meeting.wav", language="vi", beam_size=1, temperature=0.0,
    vad_filter=True, condition_on_previous_text=False)

When Whisper makes up text: silence and noise

The openai/whisper-small model card warns that the output may contain text that was never spoken in the audio, and that accuracy is lower for low-resource languages, with a higher tendency to repeat words. I tested this with experiments, starting with 20 seconds of absolute silence and 20 seconds of white noise (extras.py, model small int8):

[1] Silence and background noise (20 seconds each), model small int8
  absolute silence           vad=False -> 11 segments: Cảm ơn các bạn đã đón tìm thấy video này. | Cảm ơn các bạn đã đón tìm thấy video này. | Cả
  absolute silence           vad=True  -> 0 segments: (empty)
  white noise amplitude 0.01 vad=False -> 0 segments: (empty)
  white noise amplitude 0.01 vad=True  -> 0 segments: (empty)

Twenty seconds of absolute silence produced 11 segments repeating the sentence "Cảm ơn các bạn đã đón tìm thấy video này" (roughly "Thank you for watching this video"), exactly the kind of video-closing line that internet training data contains in huge numbers. With vad_filter=True there are no segments at all. Low-amplitude white noise does not produce text, but background music and distant speech are cases I did not measure. The practical recommendation: always turn VAD on for recordings with long silences, and filter out results with a high no_speech_prob or a phrase repeated many times.

The second experiment adds white noise to each sentence at a decreasing signal-to-noise ratio (SNR), comparing base with small, beam=1:

  SNR           base     small
  clean        20.9%     10.4%
  20 dB        23.5%     13.0%
  10 dB        30.4%     23.5%
  5 dB         47.0%    106.1%
  0 dB         67.0%     50.4%

The first two observations: WER rises steadily as noise increases, and small beats base from clean down to 10 dB. The 106.1% for small at 5 dB can exceed 100% because WER also counts inserted extra words. To verify the cause instead of guessing, I also printed the number of words the model generated at 5 dB and the raw transcript of the sentence with the largest excess:

[2b] Raw transcript at 5 dB: reference word count versus generated word count (per sentence)
  base   reference words=115 | generated words=114 | per sentence: [19, 19, 19, 16, 18, 23]
  small  reference words=115 | generated words=183 | per sentence: [19, 90, 19, 16, 17, 22]
  small, sentence s2 at 5 dB, 90 words (reference 19 words):
    họ thông thủ hành hệ thủy hệ thủy hệ thủy hệ thủy hệ thủy hệ thủy hệ thủy hệ thủy hệ thủy ... (truncated, the real output is longer)

A single sentence (s2) sent small into a "hệ thủy" loop and generated 90 words for a 19-word reference; the other five sentences have word counts close to the reference. This is exactly the repetition behavior the model card warns about, and it explains why WER passes 100% without the model mishearing many sentences.

The third observation: at 0 dB, small does better than at 5 dB (50.4% versus 106.1%). This is a reminder that with six sentences and one noise seed, the result is dominated by a few broken sentences and should not be read as a smooth curve. The operational message: below 10 dB SNR, a small Whisper model on CPU is no longer trustworthy, so you need to denoise the input, block repeated output, or use a larger model.

Note: the clean base WER here is 20.9% while the first experiment gave 14.8%, because here beam=1 and condition_on_previous_text=False. Always record the configuration next to every number.

Timestamps for subtitles

faster-whisper returns timestamps per segment, and per word when you enable word_timestamps=True (each word has start, end and word). The result for the first sentence:

  segment [ 0.00 ->  5.46] Xin chào quý khách, đơn hàng của anh đã được xác nhận và sẽ giao trong 2 ngày tới.
  words: Xin@0.0 chào@0.2 quý@0.6 khách,@0.9 đơn@1.4 hàng@1.6 của@1.9 anh@2.2 đã@2.4 được@2.7 xác@2.9 nhận@3.3 và@3.5 sẽ@3.7 giao@4.0 trong@4.3 2@4.6 ngày@4.9 tới.@5.1

Word timestamps are alignment estimates computed after decoding, so an error of a few tenths of a second is normal.

Resources when processing many files

The noise experiment process (loading both base and small int8) peaks at about 1.46 GB RSS according to /usr/bin/time -l; the configuration benchmark process (two small models) peaks at about 2.25 GB, still under the 3 GB threshold. Decoding is CPU-heavy, so set cpu_threads explicitly and push jobs into a queue instead of running them inside a web request. When running in containers or Kubernetes, set CPU and memory limits for CPU-heavy services like this one; the sizing method is covered in Requests and limits, QoS and nodeAffinity: placing pods on the right nodes.

When not to use local Whisper

• You need high accuracy for regional Vietnamese, local accents, or overlapping speakers: evaluate on your own data; you may need a larger model or a specialized service.
• You need real-time recognition with latency under one second: Whisper works on 30-second windows and is not designed for streaming; chunking techniques exist but are more complex.
• You need speaker separation (diarization): Whisper does not do this; you must add another tool.

Legal note: call recordings contain personal data and require notice and a lawful basis. Running locally keeps data inside your infrastructure but does not replace that obligation.

Common mistakes

1. Timing without consuming the generator of faster-whisper: the time is near zero because nothing has been decoded yet.
2. Comparing WER without normalization: punctuation, capitalization, digits and numbers written as words can skew WER by tens of percentage points.
3. Not using VAD on audio with long silences: it produces repeated "made-up" sentences (VAD is there to block this error, not to speed things up).
4. Assuming int8 is always faster, or drawing conclusions from a single measurement: on the M4 at beam=1, three runs showed int8 and float32 do not differ significantly in speed; the sure advantage of int8 is memory. Always include a warm-up pass and repeat the measurement several times.

Checklist for putting Whisper into a product

• Transcribe a few dozen minutes of real recordings by hand (with permission) as the reference and measure WER after normalization.
• Choose the model size by your own WER and RTF; record the beam size, compute_type and VAD setting too.
• Turn on vad_filter, set language, and block repeated output.
• Limit RAM and CPU threads, and have a clear policy for storing recordings.

The code is in code/s7-29 together with the *_output.txt files and requirements.txt; the models download automatically from Hugging Face on first use. The part that uses the say -v Linh voice only runs on macOS.

Next article

OCR for Vietnamese documents: Tesseract, PaddleOCR and post-processing. After turning speech into text, the next article handles documents stored as images: recognizing text on invoices and paperwork with Tesseract and PaddleOCR, then post-processing to fix Vietnamese diacritic errors.