vLLM is an open-source engine for serving large language models (LLMs) at high throughput. It came out of a research group at UC Berkeley and was published through the PagedAttention paper at SOSP 2023. Two ideas make it work: continuous batching (merging requests at the level of each token-generation step) and block-based KV cache management, modeled on an operating system's virtual memory. This article does not just say "vLLM is fast." It rebuilds each mechanism at small scale: measuring the real size of the KV cache, measuring why a large batch is cheaper per token, simulating scheduling, and writing a block allocator with prefix caching.

This article follows Inference optimization: ONNX export, quantization and CPU execution and Serving AI models with FastAPI. Those articles covered small models on CPU. Here we move to text-generation models, where memory, not compute, decides how many people you can serve at once. Tokens and context are explained in How LLMs work: tokens, context, sampling and limitations.

An honest limit of this article: vLLM mainly targets GPUs. The test machine is an Apple M4 with 16 GB and no NVIDIA GPU; according to the vLLM documentation, Apple Silicon support is experimental and requires building from source. So this article does not run real vLLM. Every number below is real output from the code in code/s7-49 (PyTorch 2.8.0, Transformers 4.56.0, the Qwen2-0.5B model, CPU with 2 threads) and from Ollama through its OpenAI-compatible endpoint. The vllm serve command for GPUs is a reference configuration taken from the documentation, clearly marked "needs a GPU, not run in this article."

Why serving an LLM is harder than serving a regular model

An LLM generates one token at a time. Each new token needs a pass through the whole model, and that pass has to read back the keys and values of every previous token. To avoid recomputing them, we store them; this memory region is called the KV cache. This has three consequences:

  • The number of steps is unknown in advance. One request may stop after 5 tokens, another may generate 500.
  • The KV cache grows with every step and takes most of the VRAM left after loading the weights.
  • The decode step is limited by memory bandwidth, not by matrix multiplication: each step reads all the weights just to produce one token per request in the batch. Putting more requests into one step is almost "free."

High throughput requires merging many requests into one step, and merging many requests requires managing the KV cache carefully; vLLM solves both.

How big the KV cache is: a formula and a real measurement

For each token, each layer stores one key vector and one value vector per KV head. The size in bytes is 2 x layers x KV heads x head_dim x bytes per element. The code below reads the real configuration of Qwen2-0.5B from the Hugging Face cache and compares it with two published Llama configurations:

def kv_bytes_per_token(layers, kv_heads, head_dim, dtype_bytes):
    # 2 = key and value
    return 2 * layers * kv_heads * head_dim * dtype_bytes

Output of step1_kv_size.py:

Qwen2-0.5B: layers 24 q_heads 14 kv_heads 2 head_dim 64
Qwen2-0.5B (config in cache): 12288 bytes/token fp16 = 12 KiB; 4096 tokens = 48 MiB; 32 requests x 4096 = 1.5 GiB
Llama-3-8B (published config: 32 layers, 8 KV heads, 128): 131072 bytes/token fp16 = 128 KiB; 4096 tokens = 512 MiB; 32 requests x 4096 = 16.0 GiB
Llama-2-7B (published config: 32 layers, 32 KV heads, 128, no GQA): 524288 bytes/token fp16 = 512 KiB; 4096 tokens = 2048 MiB; 32 requests x 4096 = 64.0 GiB

For models of 7 to 8 billion parameters, 32 requests of 4,096 tokens already need 16 GiB of KV cache (Llama-3-8B) up to 64 GiB (Llama-2-7B). That is far beyond a 24 GB GPU after subtracting about 16 GB of weights. Thanks to grouped-query attention (GQA, fewer KV heads than query heads), the KV cache of Llama-3-8B is four times smaller than that of Llama-2-7B at the same size: the choice of model architecture directly affects serving cost.

Does the formula match reality? step2_real_kv.py runs Qwen2-0.5B in bf16 on CPU, takes past_key_values and adds up the sizes of the tensors:

  16 tokens: measured 196608 bytes, formula 196608 bytes, match=True
  64 tokens: measured 786432 bytes, formula 786432 bytes, match=True
 256 tokens: measured 3145728 bytes, formula 3145728 bytes, match=True

An exact match, and growth is linear in length. For capacity planning, take the VRAM, subtract the weights, and divide by the KV bytes per token.

Batching: why merging requests lowers the cost per token

step3_decode_batch.py measures the time of one decode step (generating 1 token for each request) with a 128-token context, batch sizes from 1 to 16, an fp32 model and 2 CPU threads, taking the median of 5 runs:

batch= 1:    39.9 ms/step,   39.9 ms/token,   25.0 token/s
batch= 2:    69.2 ms/step,   34.6 ms/token,   28.9 token/s
batch= 4:    71.4 ms/step,   17.9 ms/token,   56.0 token/s
batch= 8:   121.4 ms/step,   15.2 ms/token,   65.9 token/s
batch=16:   135.6 ms/step,    8.5 ms/token,  118.0 token/s

Batch 16 is only about 3.4 times slower per step than batch 1 but generates 16 times more tokens, so throughput rises almost 5 times. The CPU numbers are noisy (batch 2 is unusually slow because the machine is shared), but the trend is clear: the fixed cost of each step is split across the whole batch.

A note on resources: this run used fp32 and a maximum batch of 16, so the peak RSS measured with /usr/bin/time -l was about 3.11 GB (3111976960 bytes), close to the roughly 3 GB ceiling for one process. If the machine has less RAM, use bf16 or cap the batch at 8.

From the 5 measurements we fit a linear cost model t(B) = a + b x B; the result is t(B) = 50.3 + 6.0 x B ms. This model is good enough to simulate scheduling in the next section.

Static batching and continuous batching

The naive approach is static batching (also called request-level batching): collect up to B requests, run the whole group until the longest request finishes generating, and only then accept a new group. Short requests that finish early still have to wait, empty slots in the batch are wasted, and new requests must wait for the entire old group.

Continuous batching (or iteration-level scheduling) schedules at the level of each step. After every step, finished requests leave the batch and waiting requests take the empty slots right away. This idea comes from the Orca paper (Yu et al., OSDI 2022); vLLM inherits it and combines it with PagedAttention.

step4_scheduler_sim.py simulates both policies with 64 requests arriving on a Poisson process (one request every 120 ms on average), generation lengths drawn from a long-tailed lognormal distribution, a maximum batch of 8, and the step cost taken from the model fitted above:

step cost: t(B) = 50.3 + 6.0*B ms (fitted from 5 measurements)
64 requests, generated tokens: median 34, max 205, total 2926
static batching    : done in   79.0 s, throughput  37.0 token/s, latency p50  29.4 s, p95  71.9 s
continuous batching: done in   40.0 s, throughput  73.2 token/s, latency p50  13.0 s, p95  29.4 s

This is a simulation, not a vLLM measurement, and it ignores the prefill phase. But it shows the mechanism: on the same hardware and with the same batch limit, continuous batching nearly doubles throughput and halves latency, because a 20-token request no longer waits for a 200-token request.

Two phases: prefill and decode

Every request has two phases that differ in nature. Prefill processes the whole prompt in one parallel pass, costs a lot of compute, and determines the time to first token (TTFT). Decode generates tokens one by one, costs memory bandwidth, and determines how fast the text appears. A long prompt that arrives while many users are decoding makes everyone stall. The fix is chunked prefill: split the prompt into pieces and mix them with decode steps. The vLLM documentation describes it this way: when enabled, prefill requests can be split according to the remainder of max_num_batched_tokens.

According to the vLLM V1 architecture write-up, the V1 scheduler treats prompt tokens and generated tokens the same way and represents each decision as a {request_id: number of tokens} mapping. Chunked prefill, prefix caching and speculative decoding therefore run on one mechanism, and chunked prefill is enabled by default when possible.

PagedAttention: the KV cache as virtual memory

Systems before vLLM usually gave each request one contiguous memory region large enough for the maximum length, because they could not know in advance how much the request would generate. The PagedAttention paper (Kwon et al., arXiv:2309.06180) points out the consequences: internal and external fragmentation, plus redundant copying, waste memory and limit the batch size. vLLM borrows the idea of paging. It splits the KV cache into fixed-size blocks (each holding the KV of a few tokens, for example 16), allocates blocks on demand, and uses a block table to map a request's logical positions to physical blocks. The attention kernel reads through this table, so blocks do not need to sit next to each other.

The authors report 2 to 4 times higher throughput than FasterTransformer and Orca at the same latency, with larger gains for longer sequences and larger models.

step5_paged_blocks.py builds a scaled-down model: KV memory that fits 8 requests under contiguous allocation at max_len = 2048, the actual lengths of 500 randomly generated requests drawn from a lognormal distribution, and 16-token blocks:

actual lengths: median 260, p95 735, configured max_len 2048
contiguous: 8 requests at once, 17% of reserved space actually used, 83% wasted
paged     : 45 requests fit at once in the same memory, waste only in the last block: 2.2%

The length distribution is my own assumption, so do not read "45 versus 8" as a vLLM figure; the mechanism is what matters. Contiguous allocation reserves space for the worst case, so most of it sits empty. Block allocation wastes at most one partly filled block at the end of each request. The memory saved turns into a larger batch, and a larger batch means higher throughput, as measured above.

Prefix caching: reusing the KV of identical prefixes

In enterprise applications, most prompts begin with the same system prompt, policy text or example set. Recomputing the KV for that part on every request is wasteful. According to the vLLM design documentation, automatic prefix caching caches only full blocks. The hash of each block is computed from the hash of the previous block, the block's token IDs, and extra values (LoRA, multimodal input hashes, cache_salt). Because the hash includes the parent block's hash, it reflects the entire prefix. Blocks that are no longer referenced go into a free queue and are evicted in LRU order.

The default hash algorithm is sha256 (since v0.11). cache_salt lets only requests with the same salt share blocks, a necessary safeguard in a multi-tenant system because the response-time difference between a hit and a miss could reveal someone else's prompt.

The code below implements the chained-hash mechanism on real tokens from the Qwen2 tokenizer:

def block_hashes(ids):
    hs, parent = [], b""
    for i in range(0, len(ids) - len(ids) % BLOCK, BLOCK):   # only full blocks are cached
        parent = hashlib.sha256(parent + str(ids[i:i + BLOCK]).encode()).digest()
        hs.append(parent.hex()[:8])
    return hs

Three customer questions sharing one Vietnamese system prompt, with different questions at the end:

prompt 183 tokens = 11 full blocks, prefix match 0 blocks (0 recomputed tokens skipped)
prompt 182 tokens = 11 full blocks, prefix match 10 blocks (160 recomputed tokens skipped)
prompt 177 tokens = 11 full blocks, prefix match 10 blocks (160 recomputed tokens skipped)

The first question must be computed in full; the next two reuse 10 of 11 blocks. The 11th block contains the start of the question itself, so it does not match. The practical lesson: put the fixed part first and the changing part (the question, user data) last. Inserting a timestamp or request ID at the start of the prompt destroys the whole cache.

Setting up vLLM on a GPU: a reference configuration

This vLLM section needs a GPU and was not run in this article; the parameter names come from the vLLM documentation. The server is started with vllm serve <model> and exposes OpenAI-compatible endpoints such as /v1/chat/completions, /v1/completions, /v1/embeddings and /v1/models. A skeleton example (change the model name and the values to fit your GPU):

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 64 \
  --enable-prefix-caching

The meaning of the parameters, according to the documentation: --max-model-len is the context length including both prompt and output (lowering it frees KV cache for more requests); --gpu-memory-utilization is the fraction of GPU memory that one instance may use; --max-num-seqs and --max-num-batched-tokens limit the number of requests and the number of tokens in one step; --kv-cache-dtype defaults to auto (follows the model's data type). Check the defaults for the version you install, because they change between releases.

On CPU, vLLM offers prebuilt wheels for Linux x86 and ARM, Docker images, and an experimental build from source on macOS Apple Silicon. If your goal is a quick trial on a personal machine, Ollama and llama.cpp are still the shorter path.

Load client: measuring TTFT and throughput through the OpenAI-compatible API

Whether the backend is vLLM, Ollama or a commercial service, a load-testing client is written the same way because the protocol is the same. step6_client_load.py opens a streaming connection to /v1/chat/completions and measures the time to the first chunk and the total throughput. By default it points at Ollama on localhost:11434/v1 and needs no real API key; for vLLM you only change BASE_URL and MODEL:

concurrency=1: median TTFT    568 ms, finished after  1.58 s, 60 chunks,  38.0 chunk/s (median of 5 runs)
concurrency=4: median TTFT   2411 ms, finished after  4.79 s, 240 chunks,  50.1 chunk/s (median of 5 runs)

The script does one warm-up run (loading the model), then measures 5 times and takes the median. On this shared machine (load average above 20 because many other processes ran at the same time), the result contradicts the naive expectation: four concurrent requests finish after 4.79 seconds versus 1.58 seconds for one request, and total throughput only rises from 38 to 50 chunks/s (repeat runs gave 38 to 67 chunks/s at 1 thread and 50 to 58 chunks/s at 4 threads, so the difference between the two levels is within the noise). The TTFT at 4 threads rises above 2 seconds because the requests queue up.

With its default configuration, Ollama does not merge requests into one large batch the way vLLM's continuous batching does; its level of parallelism depends on its settings and the machine's resources.

This is exactly why you measure instead of guessing: single-thread and multi-thread numbers on different backends lead to different conclusions. (The unit is a streaming chunk, approximately a token.) When you evaluate real vLLM, measure three metrics: TTFT, the time between two tokens, and total throughput at several concurrency levels, on a load whose length distribution resembles your product, and repeat several times and take the median.

When to self-host with vLLM and when not to

Self-host when traffic is steady, many users are concurrent, or data must not leave your infrastructure. Choose a commercial API or Ollama when you only have a few requests per minute (an idle GPU still costs money), when you have no GPU, or when a hard task needs quality that small open models do not reach yet: measure on your own test set.

Self-hosting vLLM brings operational work: version updates (which move fast), GPU monitoring, and endpoint security. Do not expose the server port to the internet without authentication; put it behind a reverse proxy, the same way as when you serve a model with FastAPI.

Common mistakes

  • Setting max-model-len too high "to be safe." The server must reserve room for at least one request of the maximum length; a small GPU will fail to start or will have very little KV cache left for batching.
  • Setting gpu-memory-utilization close to 1.0 when running several processes on one GPU. The parameter applies per instance, so two instances that each claim 0.9 will fight over memory.
  • Benchmarking vLLM with fixed prompts of uniform length or with a client that sends requests sequentially: the results look artificially good because there is no long tail.

Deployment checklist

  • Compute the KV bytes per token from the model configuration and derive the maximum number of concurrent tokens from the VRAM.
  • Choose a model with GQA if serving cost matters to you.
  • Set vLLM's max-model-len to your real needs, not to the model's theoretical limit.
  • Put the fixed part of the prompt first to benefit from prefix caching; use cache_salt when several customers share one server.
  • Measure vLLM with TTFT, inter-token latency and throughput at several concurrency levels, using a load close to production.

To rerun: install requirements.txt into a venv and run step1 through step6 in code/s7-49; the step*_output.txt files hold the pasted output. Original sources: the PagedAttention paper, the Orca paper (OSDI 2022), the vLLM prefix caching design, the engine arguments and vLLM installation on CPU.

Next article

MLOps with MLflow: experiment tracking, the model registry and deployment. Once a model has a place to run, the next question is lifecycle management: which version is running, which data it was trained on, and how it measured up.