Q4_K_M vs Q5_K_M: VRAM, Quality and Speed
Q5_K_M uses about 17% more memory than Q4_K_M (5.69 vs 4.85 bits per weight). On Qwen 3 32B with an 8K context that is 24.6 GB versus 21.4 GB. Use Q5_K_M when it fits with the context you need, and Q4_K_M when it does not.
VRAM on popular models
Total memory with an 8K context and FP16 KV cache. The last column is the smallest standard VRAM size that holds each version.
| Model | Q4_K_M | Q5_K_M | Difference | Smallest tier |
|---|---|---|---|---|
| Llama 3.1 8B | 6.3 GB | 7.1 GB | +0.8 GB | 8 GB / 8 GB |
| Qwen 3.5 9B | 6.4 GB | 7.3 GB | +0.9 GB | 8 GB / 8 GB |
| Qwen 3 14B | 10.5 GB | 11.9 GB | +1.4 GB | 12 GB / 16 GB |
| Mistral Small 3.1 24B | 15.6 GB | 18.0 GB | +2.3 GB | 20 GB / 20 GB |
| Qwen 3.6 27B | 16.9 GB | 19.5 GB | +2.6 GB | 20 GB / 24 GB |
| Gemma 4 31B | 19.8 GB | 22.8 GB | +3.0 GB | 24 GB / 32 GB |
| Qwen 3 32B | 21.4 GB | 24.6 GB | +3.2 GB | 24 GB / 32 GB |
| Qwen 3.6 35B-A3B | 21.0 GB | 24.4 GB | +3.4 GB | 24 GB / 32 GB |
| Llama 3.3 70B | 43.2 GB | 50.1 GB | +6.9 GB | 48 GB / 64 GB |
| Qwen 3.5 122B-A10B | 70.2 GB | 82.2 GB | +11.9 GB | 80 GB / 96 GB |
Largest model each VRAM size can run
The biggest model released since 2025 that fits fully at 8K context.
Context you can fit
Longer contexts need room for the KV cache. The longest context that fits fully at each format, with an FP16 cache:
| Model | GPU | Q4_K_M | Q5_K_M |
|---|---|---|---|
| Qwen 3 14B | 16 GB | 37K | 27K |
| Qwen 3.6 27B | 24 GB | 98K | 56K |
| Qwen 3 32B | 24 GB | 12K | Does not fit |
| Gemma 4 31B | 32 GB | 139K | 100K |
| Llama 3.3 70B | 48 GB | 14K | Does not fit |
Download size
File sizes in decimal GB, as Hugging Face shows them. Use the GGUF size calculator for any other model.
| Model | Q4_K_M | Q5_K_M |
|---|---|---|
| Llama 3.1 8B | 4.9 GB | 5.7 GB |
| Qwen 3 14B | 9.0 GB | 10.5 GB |
| Qwen 3.6 27B | 16.3 GB | 19.1 GB |
| Qwen 3 32B | 19.9 GB | 23.3 GB |
| Llama 3.3 70B | 42.8 GB | 50.2 GB |
Quality
In llama.cpp's own reference measurements on a 7B model, Q4_K_M raises perplexity over FP16 by about 0.05 and Q5_K_M by about 0.014, so Q5_K_M removes roughly three quarters of the measurable loss. In everyday chat the difference is hard to notice; it shows up more in long reasoning chains, code and non-English text, and it matters more for small models than for large ones.
Speed
Text generation reads every weight once per token, so smaller formats generate faster on the same GPU. Estimated on an RTX 4090 24GB:
| Model | Q4_K_M | Q5_K_M |
|---|---|---|
| Llama 3.1 8B | ~117 tok/s | ~101 tok/s |
| Qwen 3 32B | ~31 tok/s | ~12 tok/s (partial CPU offload) |
Verdict
Use Q5_K_M when it fits with the context you need, and Q4_K_M when it does not. The jump is about 17% more memory for a small but real quality gain. Running out of memory or offloading layers to the CPU costs far more than the quality difference, so never pick Q5_K_M if it forces offloading.
Frequently asked questions
Is Q5_K_M noticeably better than Q4_K_M?
Slightly. Benchmarks show a small, consistent improvement, but most people cannot tell the two apart in normal chat. The gap is larger on models under 8B.
Is Q5_K_M slower than Q4_K_M?
A little. Text generation is limited by memory bandwidth, so speed falls roughly in proportion to size: Q5_K_M is about 15% slower than Q4_K_M on the same GPU.
How much more VRAM does Q5_K_M need than Q4_K_M?
About 17% more for the weights (5.69 vs 4.85 bits per weight). For Qwen 3 32B with an 8K context that is 24.6 GB versus 21.4 GB.