FP16 / BF16 vs Q8_0: VRAM, Quality and Speed
FP16 / BF16 uses about 88% more memory than Q8_0 (16 vs 8.5 bits per weight). On Qwen 3 32B with an 8K context that is 64.0 GB versus 35.3 GB. There is almost no reason to run FP16 or BF16 for inference on a GPU with limited memory.
VRAM on popular models
Total memory with an 8K context and FP16 KV cache. The last column is the smallest standard VRAM size that holds each version.
| Model | FP16 / BF16 | Q8_0 | Difference | Smallest tier |
|---|---|---|---|---|
| Llama 3.1 8B | 16.8 GB | 9.8 GB | −7.0 GB | 20 GB / 12 GB |
| Qwen 3.5 9B | 18.0 GB | 10.2 GB | −7.8 GB | 20 GB / 12 GB |
| Qwen 3 14B | 29.7 GB | 16.8 GB | −12.9 GB | 32 GB / 20 GB |
| Mistral Small 3.1 24B | 46.8 GB | 25.8 GB | −21.0 GB | 64 GB / 32 GB |
| Qwen 3.6 27B | 51.8 GB | 28.3 GB | −23.5 GB | 64 GB / 32 GB |
| Gemma 4 31B | 59.7 GB | 32.9 GB | −26.8 GB | 64 GB / 48 GB |
| Qwen 3 32B | 64.0 GB | 35.3 GB | −28.6 GB | 80 GB / 48 GB |
| Qwen 3.6 35B-A3B | 66.4 GB | 35.9 GB | −30.6 GB | 80 GB / 48 GB |
| Llama 3.3 70B | 135 GB | 73.2 GB | −61.6 GB | Multi-GPU / 80 GB |
| Qwen 3.5 122B-A10B | 229 GB | 122 GB | −107 GB | Multi-GPU / Multi-GPU |
Largest model each VRAM size can run
The biggest model released since 2025 that fits fully at 8K context.
Context you can fit
Longer contexts need room for the KV cache. The longest context that fits fully at each format, with an FP16 cache:
| Model | GPU | FP16 / BF16 | Q8_0 |
|---|---|---|---|
| Qwen 3 14B | 16 GB | Does not fit | Does not fit |
| Qwen 3.6 27B | 24 GB | Does not fit | Does not fit |
| Qwen 3 32B | 24 GB | Does not fit | Does not fit |
| Gemma 4 31B | 32 GB | Does not fit | Does not fit |
| Llama 3.3 70B | 48 GB | Does not fit | Does not fit |
Download size
File sizes in decimal GB, as Hugging Face shows them. Use the GGUF size calculator for any other model.
| Model | FP16 / BF16 | Q8_0 |
|---|---|---|
| Llama 3.1 8B | 16.1 GB | 8.5 GB |
| Qwen 3 14B | 29.6 GB | 15.7 GB |
| Qwen 3.6 27B | 53.8 GB | 28.6 GB |
| Qwen 3 32B | 65.6 GB | 34.8 GB |
| Llama 3.3 70B | 141.2 GB | 75.0 GB |
Quality
Q8_0 stores each block of 32 weights as 8-bit integers with one FP16 scale, about 8.5 bits per weight. The measured difference from FP16 is around 0.0004 perplexity on llama.cpp's reference model, well below run-to-run noise in most benchmarks.
Speed
Text generation reads every weight once per token, so smaller formats generate faster on the same GPU. Estimated on an RTX 4090 24GB:
| Model | FP16 / BF16 | Q8_0 |
|---|---|---|
| Llama 3.1 8B | ~39 tok/s | ~71 tok/s |
| Qwen 3 32B | ~0.94 tok/s (partial CPU offload) | ~2.9 tok/s (partial CPU offload) |
Verdict
There is almost no reason to run FP16 or BF16 for inference on a GPU with limited memory. Q8_0 halves the weights with no meaningful quality loss and generates text nearly twice as fast. Keep FP16 for fine-tuning, for research that needs exact reproducibility, or for runtimes that do not support 8-bit formats.
Frequently asked questions
Does Q8_0 lose quality compared with FP16?
Measurably, but only barely. It is the closest GGUF format to the original weights, and it is the usual stand-in for FP16 in quality comparisons.
Is BF16 the same as FP16 for VRAM?
Yes. Both use 2 bytes per weight, so they need the same memory. BF16 has more range and less precision; most models are released in BF16.
How much more VRAM does FP16 / BF16 need than Q8_0?
About 88% more for the weights (16 vs 8.5 bits per weight). For Qwen 3 32B with an 8K context that is 64.0 GB versus 35.3 GB.