KV Cache Calculator: How Much VRAM Does Context Use?
See how much memory a model's context uses at any length and cache precision. Llama 3.1 8B stores 128 KB per token in FP16: 4.0 GB at 32K and 16.0 GB at 128K.
4.0 GB KV cache
Llama 3.1 8B at 32K tokens with the cache in FP16: 128 KB per token.
Formula: 2 (keys and values) × 32 layers × 8 KV heads × 128 head dimension × bytes per value × tokens. Grouped-query attention shares each KV head across 4 query heads, cutting the cache 4×.
| Context | FP16 | Q8_0 | Q4_0 |
|---|---|---|---|
| 2K | 0.3 GB | 0.1 GB | 72 MB |
| 4K | 0.5 GB | 0.3 GB | 0.1 GB |
| 8K | 1.0 GB | 0.5 GB | 0.3 GB |
| 16K | 2.0 GB | 1.1 GB | 0.6 GB |
| 32K | 4.0 GB | 2.1 GB | 1.1 GB |
| 64K | 8.0 GB | 4.3 GB | 2.3 GB |
| 128K | 16.0 GB | 8.5 GB | 4.5 GB |
The KV cache formula
For each token, every layer stores a key and a value vector for each KV head:
KV bytes = 2 × layers × KV heads × head dim × bytes per value × tokens
Three architecture features change this a lot:
- Grouped-query attention (GQA): most modern models share each KV head across several query heads. Llama 3 70B has 64 query heads but only 8 KV heads, an 8× saving.
- Sliding-window attention: Gemma 3 and gpt-oss limit most layers to a short window (1,024 or 128 tokens), so only a few layers grow with context.
- Multi-head Latent Attention (MLA): DeepSeek V3 and R1 cache a compressed 576-value latent per layer instead of full keys and values.
Should you quantize the KV cache?
A Q8_0 cache (llama.cpp --cache-type-k q8_0 --cache-type-v q8_0, Ollama OLLAMA_KV_CACHE_TYPE=q8_0) halves cache memory with very small quality loss in most tests. Q4_0 saves more but can hurt long-context recall. Both need flash attention enabled.
Frequently asked questions
How much VRAM does 128K context need?
It depends on the architecture. Llama 3.1 8B needs 16.0 GB for the cache alone in FP16; Gemma 3 12B needs only 8.3 GB because of its sliding window.
Does the KV cache grow during a conversation?
In llama.cpp and Ollama the full context you set is reserved at load time. vLLM reserves a pool and fills it as sequences grow. Either way, plan for the maximum context you configure.
Do parallel requests multiply the KV cache?
Yes. Each sequence needs its own cache, so 4 parallel users at 8K need as much cache as one user at 32K.