LLMs (large language models) look like a person answering questions, but at their core they do one thing: they look at the sequence of tokens so far and estimate the probability of the next token. Once you understand that mechanism, you can explain why LLMs miscount letters, why the same question gives two different answers, and why pasting more documents into a prompt sometimes makes the answer worse. This article opens the group on text-generating AI and walks through four concepts: tokens, logits and sampling, the context window, and the practical limits of LLMs.
It follows Transformer and attention: understanding the architecture through code and Fine-tuning PhoBERT with Hugging Face for Vietnamese sentiment analysis. Every number is real output from an Apple M4 with 16 GB, Python 3.12, PyTorch 2.4.0, Transformers 4.44.0, tiktoken 0.7.0 and Ollama running qwen2:1.5b. A small model has weaker absolute numbers than a commercial one, but the mechanism is identical.
What LLMs actually do: predict the next token
Today's LLMs are autoregressive models built on the Transformer decoder architecture. Each step of text generation involves five things: (1) the tokenizer turns text into a sequence of integers; (2) the Transformer network produces a vector of logits whose length equals the vocabulary size; (3) softmax turns the logits into a probability distribution; (4) a decoding algorithm picks one token from that distribution; (5) the token is appended to the input and the loop runs again until it hits an end token or the length limit. There is no "think first, then write" step: a 200-token answer is the result of 200 such iterations.
The consequence is that an LLM does not "know" what it is about to say at token 10 while writing token 1, and output quality depends on two separate things: the probability distribution (determined by training) and how a token is chosen from that distribution (determined by your configuration).
Tokens and tokenizers: the unit LLMs actually read
LLMs do not read characters or words. They read tokens, pieces of text that sit between characters and words. According to the Hugging Face Tokenization algorithms documentation, the three common algorithms are BPE, Unigram and WordPiece. BPE (byte pair encoding, Sennrich et al., 2015) starts from characters and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches the desired size. The byte-level BPE variant uses the 256 byte values as its base alphabet, so any Unicode string can be encoded without an "unknown" token (<unk>); GPT-2 uses this approach with a vocabulary of 50,257 tokens.
For anyone shipping LLMs in a Vietnamese product, the point to remember is that the vocabulary is learned from training data, so languages that appear less often get cut into more pieces. The tokens_demo.py script counts the tokens of four sample sentences with three OpenAI encodings through tiktoken (gpt2, cl100k_base from GPT-4, o200k_base from GPT-4o) and the Qwen2 tokenizer.
[Tokens / characters] (characters counted as Unicode, not bytes)
Type chars bytes gpt2 cl100k o200k qwen2
English 57 57 11 12 12 15
Vietnamese 62 76 45 27 18 23
Code 64 64 25 17 17 17
Numbers 16 16 7 6 6 16The 62-character Vietnamese sentence needs 45 tokens with gpt2, 27 with cl100k_base, 23 with Qwen2 and 18 with o200k_base, while the 57-character English sentence needs only 11 to 15. With cl100k_base, the Vietnamese version costs more than twice as many tokens as the English one, which feeds into the API bill, latency and the usable context. Looking at the exact split shows why:
[How cl100k_base splits the Vietnamese sentence]
[b'V', b'ui', b' l\xc3\xb2ng', b' x', b'\xc3\xa1c', b' nh', b'\xe1\xba\xadn', b' than', b'h', b' to', b'\xc3\xa1n', b' cho', b' \xc4\x91', b'\xc6\xa1n']
[How Qwen2 splits the Vietnamese sentence]
['V', 'ui', 'Ġlòng', 'Ġxác', 'ĠnháºŃn', 'Ġthan', 'h', 'Ġtoán', 'Ġcho', 'ĠÄijÆ¡n', 'ĠhÃłng', 'Ġ', '2', '0', '4', '8', '1', 'ĠtrÆ°á»Ľc', 'Ġthứ', 'ĠS', 'á', 'u', '.']Vietnamese diacritics get split into separate byte pieces (nh + ận, x + ác); the Qwen2 tokenizer shows the mapped byte-level form (Ġ is a space, ò is the two bytes of the letter ò). The lesson for LLMs: count tokens with the exact tokenizer of the model you call, and never estimate from word or character counts.
Three more details in the output. Numbers are split differently: cl100k_base groups three digits per token while Qwen2 splits every digit, so a 16-digit string costs 6 and 16 tokens, which affects arithmetic ability. order, Order, ORDER and order with a leading space are four different tokens, so stray whitespace changes the input. The vocabularies have 50,257 (gpt2), 100,277 (cl100k_base), 200,019 (o200k_base) and 151,646 (Qwen2) tokens.
Tokens also explain a familiar "silly" LLM mistake: the word strawberry is seen as three pieces (str + aw + berry with cl100k_base), not as a string of letters, so counting letters is something the model has to learn indirectly.
From logits to probabilities: LLM temperature
After the Transformer processes the prompt, the output at the last position is the logits vector, with one element per token in the vocabulary. Softmax converts it to probabilities. Temperature T divides the logits before softmax: a small T sharpens the distribution, a large T flattens it, and T approaching 0 is equivalent to always choosing the most probable token. The sampling_demo.py script takes the logits straight from Qwen2-0.5B with the prompt "Thủ đô của Việt Nam là" ("The capital of Vietnam is"), then implements softmax itself with NumPy.
[T=1.0] top-6: ' Hà':0.248, ' Thủ':0.096, ' thủ':0.065, ' sa':0.037, ':\n':0.032, ' Ho':0.028
entropy = 4.27 nat | sum of top-6 probabilities = 0.506
[T=0.2] top-4: ' Hà':0.990, ' Thủ':0.009, ' thủ':0.001, ' sa':0.000 | entropy = 0.06
[T=0.7] top-4: ' Hà':0.526, ' Thủ':0.135, ' thủ':0.079, ' sa':0.034 | entropy = 2.12
[T=1.5] top-4: ' Hà':0.049, ' Thủ':0.026, ' thủ':0.020, ' sa':0.014 | entropy = 8.13At T = 1.0 the model puts 24.8% on Hà (the start of "Hà Nội"), and the top six tokens cover only 50.6%: the rest sits in a long "tail" of tens of thousands of tokens. Lowering T to 0.2 pushes Hà up to 99.0% (entropy 0.06); raising T to 1.5 drops it to 4.9% (entropy 8.13). For LLMs like this one, the logits size (151,936) is larger than the real vocabulary (151,646) because the embedding table is rounded up.
Sampling in LLMs: greedy, top-k and top-p
LLMs have three common ways to choose a token, and each one changes the output. Greedy always takes the highest-probability token; according to the Hugging Face Generation strategies documentation, greedy suits short outputs but starts repeating itself on long text. Sampling (multinomial) draws at random from the distribution, so every token with nonzero probability has a chance. Beam search keeps several candidate sequences and picks the one with the highest total probability, which suits tasks that stay close to the input, such as speech recognition or image captioning.
Pure sampling that lands in the tail of the distribution, full of meaningless tokens, sends the whole sentence off course. The paper The Curious Case of Neural Text Degeneration (Holtzman et al., ICLR 2020) shows that likelihood-maximizing decoding produces bland, repetitive text, and proposes nucleus sampling (top-p): sample only from the smallest set of tokens whose cumulative probability reaches p. Top-k is simpler: keep the k highest tokens. Here are both implemented:
def top_k_filter(p, k):
q = p.copy(); q[np.argsort(-p)[k:]] = 0; return q / q.sum()
def top_p_filter(p, top_p):
order = np.argsort(-p); cum = np.cumsum(p[order])
cut = np.searchsorted(cum, top_p) + 1 # smallest number of tokens whose cumulative probability >= top_p
q = np.zeros_like(p); q[order[:cut]] = p[order[:cut]]; return q / q.sum(), cut
[Tokens remaining after filtering]
top_k=40 -> 40 tokens
top_p=0.5 -> 6 tokens
top_p=0.9 -> 205 tokens
top_p=0.95 -> 621 tokensTop-k keeps a fixed 40 tokens whether the distribution is sharp or flat, while top-p adapts: p = 0.5 keeps 6 tokens, p = 0.9 keeps 205, p = 0.95 keeps 621. Drawing 2,000 samples from each distribution and counting the distinct tokens that appear gives:
[2,000 draws of the next token: distinct tokens seen, most frequent token]
T=1.0, no filter : (338, ' Hà')
T=1.0, top_k=40 : (40, ' Hà')
T=1.0, top_p=0.9 : (158, ' Hà')
T=0.2 : (3, ' Hà')
T=1.5, no filter : (1184, ' Hà')With no filter at T = 1.0 you get 338 distinct tokens; top-k = 40 leaves 40; top-p = 0.9 leaves 158; T = 0.2 leaves only 3; T = 1.5 with no filter gives 1,184. That is why the practical advice is to adjust one of the two knobs (temperature or top-p), not both.
Finally, let LLMs generate real text with the Transformers model.generate method and 14 new tokens:
[Generate 14 tokens]
greedy : Hà Nội. Hà Nội là thủ đô của Việt Nam, là một
greedy (run 2): Hà Nội. Hà Nội là thủ đô của Việt Nam, là một
T=1.0 seed=1: :\nTrại Quốc gia \n* Thành phố Hồ Chí Minh\n
T=1.0 seed=2: thủ đô capital Thủ đô tại Trung Quốc mở rộng, được gọi
T=1.0 seed=3: : Paris | Tổng hợp\nNhằm phân tích thị trường Nhà nước
T=1.5 seed=1: :\nTrại Pierre奇督囍voir fi 필ופ پ.perform
T=1.5 seed=2: thủ phủ quê tắtỏi tại kênh L序列 mở.\n yüz compens foreverGreedy gives one answer, identical across two runs. Sampling at T = 1.0 gives three different results, some of them off topic (Paris, Trung Quốc, meaning China); at T = 1.5 the text shatters into fragments of several languages. Sampling in LLMs always trades consistency for diversity.
Greedy or temperature = 0 is repeatable on your own machine, but with commercial APIs it is not absolutely guaranteed because of parallel GPU computation, batching and model updates. Do not test by comparing exact strings; the article Systematic prompt engineering covers criteria-based testing.
Context window: how much LLMs can remember
The context window is the total number of tokens LLMs process in one call, including both the prompt and the generated output. LLMs have no memory between calls: a "conversation" is the client resending the entire history every turn. According to the specifications the providers had published as of August 2024, GPT-4o and Llama 3.1 have a 128K-token window, Claude 3.5 Sonnet has 200K, and Gemini 1.5 Pro has 1 to 2 million (the 2 million version opened to developers in June 2024); the Qwen2-1.5B used here has 32,768 (read from the metadata through Ollama).
A long window is not free: LLMs store the keys and values of every previous token at every layer, called the KV cache. Each token costs 2 (key and value) × number of layers × number of KV heads × head_dim × bytes per value. Qwen2-1.5B has 28 layers, 12 query heads but only 2 KV heads thanks to grouped-query attention (GQA), and head_dim = 128:
[KV cache] per token = 2 x 28 layers x 2 KV heads x 128 x 2 bytes = 28,672 bytes
4096 tokens -> 112.0 MiB
32768 tokens -> 896.0 MiB32,768 tokens take 896 MiB; with all 12 KV heads it would be 5,376 MiB. That is why many newer LLMss use GQA. Prompt processing time also grows with length; the total time on Ollama (processing the prompt and generating up to 40 tokens) is:
[Total time by prompt length]
312 tokens | 1.2s
1187 tokens | 1.8s
5874 tokens | 8.2sLong does not mean well used: lost in the middle
Fitting inside the window does not mean LLMs can use everything. The paper Lost in the Middle (Liu et al., TACL 2023) shows that performance is usually highest when the information sits at the beginning or end of the context and drops clearly when it sits in the middle. The experiment: a list of 200 lines like "Order DH-7xxx has confirmation code ..., ships from warehouse ..." (written in Vietnamese; 5,891 tokens), then ask for the code of an order in different regions (8 questions per region, temperature = 0).
[Lookup in a 200-line list, num_ctx=8192, 8 different lines asked per position]
region 0% | prompt 5891 tokens | correct 7/8
region 25% | prompt 5891 tokens | correct 2/8
region 50% | prompt 5891 tokens | correct 1/8
region 75% | prompt 5891 tokens | correct 1/8
region 95% | prompt 5891 tokens | correct 1/8
The first region scores 7/8, the 25% region drops to 2/8, and the later regions reach only 1/8. The end of the list does not recover like the U-shaped curve in the paper, because a small model is weaker at lookup in general; a sample of 8 questions per region is also very small. The rule to take away: put important information at the beginning or end of the prompt, and for long documents, retrieving exactly the passage you need usually beats pasting everything.
The num_ctx trap: truncating the prompt without an error
The Ollama FAQ describes the default window as only a few thousand tokens, changed with the num_ctx parameter; the default value changes between versions, so set it explicitly. The outputs below ran on Ollama 0.40.15 (taken from the /api/version endpoint); the truncation behavior here was observed on that version, and older Ollama releases may differ. When the prompt is longer than num_ctx, Ollama raises no error and drops the beginning instead. Asking about 8 orders at the start of the list with three num_ctx values:
[num_ctx smaller than the prompt: ask about the FIRST 8 lines of the list]
num_ctx= 8192 | prompt_eval_count= 5891 | correct 7/8
num_ctx= 2048 | prompt_eval_count= 1026 | correct 0/8
num_ctx= 1024 | prompt_eval_count= 514 | correct 0/8With 8192, the model sees all 5,891 tokens and gets 7/8. With 2048 and 1024, prompt_eval_count is only 1,026 and 514 tokens, so more than 80% of the prompt was dropped before the model saw it, and the result is 0/8 with no error reported. A defense for applications built on LLMs: compare prompt_eval_count (or usage.prompt_tokens in other APIs) with your precomputed token count and warn when they differ.
Limits of LLMs you have to design around
The mechanism above implies a fairly stable list of limits for every LLM, large or small.
- Hallucination (making things up): LLMs are optimized to produce high-probability text, not correct text. You need a source to check against (RAG), verification in code, and room for the model to answer "I don't know".
- Knowledge stops at the training cutoff: LLMs do not know today's prices or your company's new policy unless you put them in the prompt.
- Nondeterminism: sampling gives different results, and results can change between model versions.
- Unreliable arithmetic and counting: because of how text is split into tokens, multi-digit calculations and character counting are error-prone. Use tools (code) for these.
- Finite and unevenly used context: beyond the window, input is truncated or rejected; inside the window, information in the middle is easily overlooked.
- Prompt injection: text from outside sources (email, web pages) sits in the same window as your instructions, and LLMs cannot reliably tell which is a command and which is data. Do not give LLMs sensitive actions without a control layer.
When to use LLMs and when not to
LLMs are strong at open-ended language work such as summarization, intent classification, extracting information from free text and answering from supplied documents. They are weak and expensive for jobs that already have clear rules: format validation, calculating money, matching order codes. Classifying millions of comments with fixed labels is usually cheaper with a small model like PhoBERT, so LLMs are not always the right tool; try the cheapest route first and measure on real data.
Common mistakes
- Estimating tokens from word counts: for Vietnamese the error is large, as the count table above shows.
- Setting a high temperature for extraction tasks, or turning both temperature and top-p at the same time.
- Pasting a long document and believing LLMs read all of it: information in the middle is easily missed.
- Not checking the actual number of tokens sent: the prompt is silently truncated when
num_ctxis small.
Checklist
- For every feature built on LLMs, count tokens with the model's own tokenizer and track both input and output tokens.
- Use greedy or a low T for extraction and classification; a moderate T with top-p for content generation.
- Set the context size explicitly and warn when the actual token count differs from the expected one.
- Hand calculation, counting and format checks to code; use LLMs for language understanding.
The source code in code/s7-31 contains three scripts tokens_demo.py, sampling_demo.py, context_demo.py, the *_output.txt files (the output pasted above) and requirements.txt. The first two scripts download Qwen/Qwen2-0.5B from the Hugging Face Hub, and the third needs Ollama with qwen2:1.5b (the Qwen2 family was chosen because it matches the time of the article, August 2024). The order data is synthetic.
Next article
Systematic prompt engineering: roles, examples and prompt testing. Now that you know LLMs only guess the next token from its context, the next article moves to writing that context with discipline: assigning roles, giving examples and measuring whether a new prompt beats the old one.
