Q4_K_M vs Q8_0: VRAM, Quality and Speed
Q8_0 uses about 75% more memory than Q4_K_M (8.5 vs 4.85 bits per weight). On Qwen 3 32B with an 8K context that is 35.3 GB versus 21.4 GB. Q8_0 needs about 75% more memory than Q4_K_M for a modest quality gain.
VRAM on popular models
Total memory with an 8K context and FP16 KV cache. The last column is the smallest standard VRAM size that holds each version.
| Model | Q4_K_M | Q8_0 | Difference | Smallest tier |
|---|---|---|---|---|
| Llama 3.1 8B | 6.3 GB | 9.8 GB | +3.4 GB | 8 GB / 12 GB |
| Qwen 3.5 9B | 6.4 GB | 10.2 GB | +3.8 GB | 8 GB / 12 GB |
| Qwen 3 14B | 10.5 GB | 16.8 GB | +6.3 GB | 12 GB / 20 GB |
| Mistral Small 3.1 24B | 15.6 GB | 25.8 GB | +10.2 GB | 20 GB / 32 GB |
| Qwen 3.6 27B | 16.9 GB | 28.3 GB | +11.4 GB | 20 GB / 32 GB |
| Gemma 4 31B | 19.8 GB | 32.9 GB | +13.0 GB | 24 GB / 48 GB |
| Qwen 3 32B | 21.4 GB | 35.3 GB | +13.9 GB | 24 GB / 48 GB |
| Qwen 3.6 35B-A3B | 21.0 GB | 35.9 GB | +14.9 GB | 24 GB / 48 GB |
| Llama 3.3 70B | 43.2 GB | 73.2 GB | +30.0 GB | 48 GB / 80 GB |
| Qwen 3.5 122B-A10B | 70.2 GB | 122 GB | +51.8 GB | 80 GB / Multi-GPU |
Largest model each VRAM size can run
The biggest model released since 2025 that fits fully at 8K context.
Context you can fit
Longer contexts need room for the KV cache. The longest context that fits fully at each format, with an FP16 cache:
| Model | GPU | Q4_K_M | Q8_0 |
|---|---|---|---|
| Qwen 3 14B | 16 GB | 37K | Does not fit |
| Qwen 3.6 27B | 24 GB | 98K | Does not fit |
| Qwen 3 32B | 24 GB | 12K | Does not fit |
| Gemma 4 31B | 32 GB | 139K | Does not fit |
| Llama 3.3 70B | 48 GB | 14K | Does not fit |
Download size
File sizes in decimal GB, as Hugging Face shows them. Use the GGUF size calculator for any other model.
| Model | Q4_K_M | Q8_0 |
|---|---|---|
| Llama 3.1 8B | 4.9 GB | 8.5 GB |
| Qwen 3 14B | 9.0 GB | 15.7 GB |
| Qwen 3.6 27B | 16.3 GB | 28.6 GB |
| Qwen 3 32B | 19.9 GB | 34.8 GB |
| Llama 3.3 70B | 42.8 GB | 75.0 GB |
Quality
Q8_0 is effectively lossless: llama.cpp's reference measurement puts it within about 0.0004 perplexity of FP16, against about 0.05 for Q4_K_M. Q4_K_M keeps most of a model's ability, but on hard reasoning, maths and code the 8-bit version makes fewer small mistakes.
Speed
Text generation reads every weight once per token, so smaller formats generate faster on the same GPU. Estimated on an RTX 4090 24GB:
| Model | Q4_K_M | Q8_0 |
|---|---|---|
| Llama 3.1 8B | ~117 tok/s | ~71 tok/s |
| Qwen 3 32B | ~31 tok/s | ~2.9 tok/s (partial CPU offload) |
Verdict
Q8_0 needs about 75% more memory than Q4_K_M for a modest quality gain. If you have room, Q8_0 of a model is the most faithful version you can run. If you do not, a larger model at Q4_K_M usually beats a smaller model at Q8_0 in the same memory.
Frequently asked questions
Should I run a 14B model at Q8_0 or a 32B model at Q4_K_M?
Usually the 32B at Q4_K_M. They need similar memory, and the larger model's extra knowledge and reasoning outweigh the quantization loss.
Is Q8_0 as good as the original model?
For practical purposes, yes. Measured differences from FP16 are tiny, which is why Q8_0 is the usual reference when people compare quantizations.
How much more VRAM does Q8_0 need than Q4_K_M?
About 75% more for the weights (8.5 vs 4.85 bits per weight). For Qwen 3 32B with an 8K context that is 35.3 GB versus 21.4 GB.