KV Cache VRAM: 8 GiB Per User at 32k on a 32B Model

📘 Tutorials 2026-11-21 6 min read

Concurrency promised from parameter count does not survive contact with the card. The model that fits is not always the model that scales.

📜 Table of Contents

Model size is the wrong number to size a GPU by. Two 7B models, same weights, same card, same context: one holds 17 concurrent 8k conversations, the other holds 8. The difference is entirely in the KV cache, and the KV cache is a function of attention heads, not parameters.

Every number below is computed from the model's own config.json and its Hugging Face safetensors total, pulled on 21 November 2026. Nothing here is a vendor benchmark. The formula is the one vLLM and Hugging Face both use: 2 (keys and values) x layers x kv_heads x head_dim x 2 bytes, per token.

What one token of context costs, per model

modellayersKV headshead dimKV per token8k, GiB32k, GiBweights, fp16, GiB
Qwen2.5-7B-Instruct28412856 KiB0.441.7514.19
Mistral-7B-Instruct-v0.3328128128 KiB1.004.0013.50
Llama-3.1-8B-Instruct328128128 KiB1.004.0014.96
Gemma-2-9B-it428256336 KiB2.62n/a (8k cap)17.21
Qwen2.5-14B-Instruct488128192 KiB1.506.0027.51
Mistral-Small-24B408128160 KiB1.255.0043.91
Qwen2.5-32B-Instruct648128256 KiB2.008.0061.03
Qwen3-32B648128256 KiB2.008.0061.02

Read the 32k column as memory per conversation. A 32B model at 32k context costs 8.00 GiB of cache for one user. Four users cost 32.00 GiB, which is more memory than a 24 GB card has in total, and the weights are not in that count yet.

Why two 7B models differ by 2.3x

Qwen2.5-7B-Instruct sets num_key_value_heads: 4 against 28 attention heads. Mistral-7B-Instruct-v0.3 keeps 8 KV heads against 32. Same parameter count to within 5%, 56 KiB versus 128 KiB of cache per token, 2.3x apart.

That is the whole trick behind grouped-query attention. The weights cost what they cost. The cache costs whatever the architect decided the KV head count should be. When you are picking a model to serve, this single field decides the concurrency you can sell.

The model where hidden size divided by heads gives the wrong answer

Most of the time head_dim = hidden_size / num_attention_heads. Gemma-2-9B-it breaks it: 3,584 hidden size, 16 heads, which suggests 224, while the config declares head_dim: 256. Compute with 224 slots and you get 294 KiB per token instead of the real 336 KiB, a 12.5% undercount that turns into a model that runs out of cache mid-afternoon.

The rule that survives contact with real configs: read head_dim if it is present, fall back to hidden_size / num_attention_heads only when it is absent.

What a 24 GB card actually holds

Assume the usual 0.9 utilisation ceiling, so 21.6 GiB of the card is addressable. Subtract the fp16 weights from the safetensors metadata, divide what is left by the per-token cost, and the concurrency falls out:

modelweightsleft for cachetokens8k streams
Qwen2.5-7B-Instruct14.19 GiB7.41 GiB138,83916.9
Mistral-7B-Instruct-v0.313.50 GiB8.10 GiB66,3518.1
Llama-3.1-8B-Instruct14.96 GiB6.64 GiB54,4156.6
Gemma-2-9B-it17.21 GiB4.39 GiB13,6871.7

Same card, same context length, 17 conversations on one and under 7 on another. Buyers comparing "7B on a 24 GB card" are comparing nothing until someone says which 7B.

The 32B case is worse than the weights suggest. On a single 80 GB card, 0.9 utilisation leaves 72 GiB, the fp16 weights take 61.03 GiB, and 10.97 GiB remains for cache. At 256 KiB per token that is 44,943 tokens, or 1.37 concurrent conversations at 32k. A single 32k chat on a 32B model at fp16 nearly fills an H100's usable memory before batch size enters the picture.

Four things that move the number

fp8 cache. vLLM accepts --kv-cache-dtype fp8 on hardware that supports it, which halves every figure in the tables. That is the cheapest concurrency you will ever buy, and it is a flag, not a purchase.

Context length is a ceiling, not a reservation. Paged attention allocates blocks as tokens arrive, so an idle server with --max-model-len 32768 does not sit on 8 GiB of cache per slot. The tables tell you what a conversation costs at length, not what the card holds at idle.

The utilisation default is not always 0.9. In vLLM 0.34.0 the resolved value is 1.0 when the extensible KV cache is on (the default on CUDA and ROCm, sized from measured memory), and 0.92 otherwise. The cache config source says to leave gpu_memory_utilization unset unless something outside vLLM needs the memory. Sizing arithmetic that assumes 0.9 is a conservative estimate, not the engine's behaviour.

Tensor parallel is not free for cache. KV heads are split across GPUs, so a 2-way split halves the cache per GPU and roughly preserves the token count per GPU, while weights also halve. The aggregate is what improves, and only if the KV head count divides evenly across devices. Eight KV heads on four GPUs means two each. Eight on three does not divide, and vLLM refuses the configuration.

Compute it for the model you are actually deploying

Two fields come from the config, one from the safetensors metadata, and the rest is arithmetic. This runs anywhere with the huggingface_hub package installed:

from huggingface_hub import hf_hub_download, HfApi
import json

repo = "Qwen/Qwen2.5-7B-Instruct"
cfg = json.load(open(hf_hub_download(repo, "config.json")))
info = HfApi().model_info(repo)

layers = cfg["num_hidden_layers"]
heads = cfg["num_attention_heads"]
kv = cfg.get("num_key_value_heads", heads)
hd = cfg.get("head_dim") or cfg["hidden_size"] // heads
params = info.safetensors.total

per_token = 2 * layers * kv * hd * 2          # bytes, fp16
weights_gib = params * 2 / 1024**3
print("KV per token: %.1f KiB" % (per_token / 1024))
print("weights: %.1f GiB" % weights_gib)
for ctx in (8192, 32768, 131072):
    print(ctx, "%.2f GiB per conversation" % (per_token * ctx / 1024**3))

On Qwen2.5-7B that prints 56.0 KiB, 14.2 GiB, and 0.44 / 1.75 / 7.00 GiB. If your number differs from the tables above, the config changed or you picked a different checkpoint variant, and the checkpoint you picked is the one that will run.

What this does not cover

Activations, CUDA graphs, the paged-attention block table, and allocator fragmentation all take memory that this arithmetic ignores. On a 24 GB card the gap is usually one to two GiB, which is why the stream counts above should be treated as ceilings to test against, not as a provisioning plan. Load the model, run nvidia-smi, compare it with the estimate, then promise concurrency.

The measured version of this same arithmetic on a small card, with Ollama, is in Ollama in Docker: the VRAM math. What the cache is and why it grows per token is in KV cache explained. If the question is which engine to put on the card once it fits, the comparison is in vLLM vs Ollama.

Related Articles
2026-11-19
LiteLLM Proxy: Per-Key Budgets and Fallbacks (v1.110.0)
2026-07-14
Free AI Coding Assistant Setup 2026: 5-Min VS Code Guide (Continue, Copilot, Windsurf)
2026-07-14
Open Source AI Model Benchmarks 2026: Llama 3.1 vs Qwen 2.5 vs Mistral vs Phi-3
2026-08-05
Object Detection with YOLO in 2026: Ultralytics (60k Stars) Tutorial - Detect Objects in Images and Video with 10 Lines
2026-08-07
AI Video Translator: Open-Source Dubbing and Subtitles That Work
2026-07-26
Self-Hosted AI Hardware Guide 2026: Choosing a Machine for Local LLMs

Written by our editorial team; tools listed here are tested or verified against public sources. Links point to official sites or GitHub repos for reference only — no paid placements.

💬 Comments (0)

No comments yet. Be the first!

Login to comment