Why does context length use so much VRAM?

Every token in the context stores a key and value vector on every attention layer. On a model like Llama 3.3 70B that is 320 KB per token, so 32K tokens need 10.0 GB.

Does quantizing the KV cache hurt quality?

Q8_0 has a negligible effect in most tests. Q4_0 saves more memory but can reduce accuracy when recalling details from long documents.

Is the KV cache stored in VRAM or RAM?

In VRAM by default, next to the layers it belongs to. If layers are offloaded to the CPU, their cache usually moves to system RAM too, unless the runtime is told to keep it on the GPU.

Why is the KV cache smaller on newer models?

Newer designs share key and value heads across many query heads, limit most layers to a sliding window, compress keys and values into a latent vector, or replace most attention layers with linear attention that keeps a fixed-size state.

KV Cache Explained

Updated October 14, 2026 · Figures calculated live from our model data

The KV cache stores keys and values for every token in the context so the model does not recompute them. It grows linearly with context length, and on older models it can exceed the weights: Llama 3.1 8B needs 16.0 GB of cache at 128K tokens. Newer designs cut that dramatically.

The formula

KV cache = 2 × layers × KV heads × head dimension × bytes per value × tokens

For Llama 3.1 8B: 2 × 32 × 8 × 128 × 2 bytes = 128 KB per token. Multiply by 32,768 tokens and you get 4.0 GB.

Same size, very different caches

Model	Attention design	KB per token	Cache at 32K	Cache at 128K
Llama 3.3 70B	GQA, 8 KV heads, 80 layers	320	10.0 GB	40.0 GB
Qwen 3 32B	GQA, 8 KV heads, 64 layers	256	8.0 GB	n/a (max 40K)
Mistral Small 3.1 24B	GQA, 8 KV heads, 40 layers	160	5.0 GB	20.0 GB
Gemma 3 27B	Sliding window on 52 of 62 layers	80	2.9 GB	10.4 GB
Gemma 4 31B	Sliding window on 50 of 60 layers	80	3.3 GB	10.8 GB
Qwen 3.6 27B	Hybrid: 16 of 64 layers attend	64	2.1 GB	8.1 GB
Qwen 3.6 35B-A3B	Hybrid MoE, 2 KV heads	20	0.7 GB	2.6 GB
DeepSeek R1 671B	Multi-head latent attention	69	2.1 GB	8.6 GB
gpt-oss-120b	Alternating 128-token window	36	1.1 GB	4.5 GB

The tricks, one by one

Grouped-query attention (GQA): many query heads share one key/value head. Llama 3 uses 8 KV heads for 32–64 query heads, a 4–8× saving.
Sliding windows: most layers only look at the last 512–1,024 tokens, so their cache stops growing. Gemma 3, Gemma 4 and gpt-oss use this.
Multi-head latent attention (MLA): DeepSeek, Kimi and Mistral Small 4 compress keys and values into one small latent vector per layer.
Hybrid linear attention: Qwen 3.5 and later replace three of every four attention layers with Gated DeltaNet layers that keep a fixed-size state instead of a growing cache.
Shared KV layers: Gemma 4 E2B and E4B reuse earlier layers' caches in their last layers.

Quantizing the cache

Runtimes can store the cache in 8 or 4 bits. Q8_0 halves it with negligible quality loss; Q4_0 quarters it but can hurt long-context recall. For Qwen 3 32B at 32K:

Cache type	Cache size	Total at Q4_K_M
FP16 (default)	8.0 GB	27.4 GB
Q8_0	4.3 GB	23.6 GB
Q4_0	2.3 GB	21.6 GB

In llama.cpp use -ctk q8_0 -ctv q8_0 -fa on; in Ollama set OLLAMA_KV_CACHE_TYPE=q8_0 with flash attention. vLLM supports an FP8 cache with --kv-cache-dtype fp8.

Batching multiplies it

Each parallel request has its own cache, so serving four users at 32K costs four times the memory above. That is why serving setups care so much about cache size, and why parallel slots are a common cause of surprise out-of-memory errors.

Try any model and context in the KV cache calculator.

Why models keep a cache at all

A language model writes one token at a time. To decide the next token, every layer looks back at every earlier token through attention, which needs that token's key and value vectors. Recomputing them for the whole conversation at every step would make generation impossibly slow, so runtimes compute them once and keep them: that is the KV cache. It trades memory for speed, and the trade is always worth it.

Worked example: the same context on four models

A 64K-token context, FP16 cache:

Model	Cache at 64K	Share of total
Llama 3.1 8B	8.0 GB	60%
Qwen3-Coder 30B-A3B	6.0 GB	25%
Gemma 4 31B	5.8 GB	24%
Qwen 3.6 27B	4.1 GB	20%

On the older Llama design the cache is a large share of the total; on the hybrid Qwen model it is small.

Prompt caching and context shifting

Runtimes reuse the cache between requests when the beginning of the prompt is the same, for example a long system prompt or the earlier turns of a chat. That makes follow-up questions fast. When the context fills up, some runtimes shift it, dropping the oldest tokens, rather than failing. Neither feature changes how much memory the cache needs; they change how often it is recomputed.

Paged attention in vLLM

Serving engines such as vLLM and SGLang manage the cache in small pages instead of one block per request. Memory is shared between requests and only allocated as text is generated, which lets one GPU serve many users. That is why vLLM reserves most of the GPU up front: the reserved space is a pool for everyone's cache.

Practical rules

Set the context to what you actually use.
Use a Q8_0 cache when you need more context; it is close to free in quality.
Prefer models with sliding-window, latent or hybrid attention for long documents.
Remember that parallel requests each need their own cache.