Qwen 2.5 Coder 32B VRAM Requirements
Qwen 2.5 Coder 32B needs about 21.4 GB of VRAM at Q4_K_M with an 8K context: 18.5 GB for weights, 2.0 GB for the KV cache and 0.9 GB of runtime overhead. At Q8_0 it needs 35.3 GB and at full FP16 precision 64.0 GB. The smallest GPU that runs it fully is the NVIDIA L4 24GB.
21.4 GB VRAM
Qwen 2.5 Coder 32B at Q4_K_M with 8K context and the KV cache in FP16.
- Weights 18.5 GB
- KV cache 2.0 GB
- Runtime overhead 0.9 GB
Which GPUs can run it
- RTX 3060 12GB53% on GPU~3.5 tok/s
- RTX 4060 Ti 16GB70% on GPU~4.4 tok/s
- RTX 3090 24GBFits~28 tok/s
- Radeon RX 7900 XTX 24GBFits~25 tok/s
- RTX 4090 24GBFits~31 tok/s
- RTX 5090 32GBFits~54 tok/s
- RTX 6000 Ada 48GBFits~29 tok/s
- A100 80GBFits~61 tok/s
- H100 80GB SXMFits~97 tok/s
- Ryzen AI Max+ 395 128GBFits~7.3 tok/s
- Apple M4 Max 128GBFits~17 tok/s
- Apple M5 Max 128GBFits~19 tok/s
- RTX PRO 6000 Blackwell 96GBFits~54 tok/s
- NVIDIA DGX Spark 128GBFits~7.8 tok/s
- Apple M3 Ultra 512GBFits~25 tok/s
Qwen 2.5 Coder 32B VRAM by quantization
Total memory with an FP16 KV cache. "Smallest tier" is the smallest standard VRAM size with a 6% safety margin at 8K.
| Quantization | Bits/weight | Weights | Total at 8K | Total at 32K | Smallest tier |
|---|---|---|---|---|---|
| Q4_K_M | 4.85 | 18.5 GB | 21.4 GB | 27.4 GB | 24 GB |
| Q5_K_M | 5.69 | 21.7 GB | 24.6 GB | 30.6 GB | 32 GB |
| Q6_K | 6.59 | 25.2 GB | 28.0 GB | 34.0 GB | 32 GB |
| Q8_0 | 8.50 | 32.5 GB | 35.3 GB | 41.3 GB | 48 GB |
| FP16 / BF16 | 16.00 | 61.1 GB | 64.0 GB | 70.0 GB | 80 GB |
| Q3_K_M | 3.91 | 14.9 GB | 17.8 GB | 23.8 GB | 20 GB |
| Q2_K | 3.00 | 11.5 GB | 14.3 GB | 20.3 GB | 16 GB |
VRAM by context length
At Q4_K_M. The KV cache grows linearly with context: 0.25 GB per 1,000 tokens in FP16.
| Context | KV cache (FP16) | KV cache (Q8_0) | Total, FP16 cache | Total, Q8_0 cache |
|---|---|---|---|---|
| 4K tokens | 1.0 GB | 0.5 GB | 20.4 GB | 19.9 GB |
| 8K tokens | 2.0 GB | 1.1 GB | 21.4 GB | 20.5 GB |
| 32K tokens | 8.0 GB | 4.3 GB | 27.4 GB | 23.6 GB |
GPUs that can run Qwen 2.5 Coder 32B
At Q4_K_M with 8K context. "Best quant" is the highest-quality format that fits fully; "max context" is at Q4_K_M.
| GPU | Usable memory | Status | Best quant | Max context | Speed |
|---|---|---|---|---|---|
| NVIDIA L4 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~9.2 tok/s |
| Tesla P40 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~9.1 tok/s |
| Apple M4 Max 36GB | 24.0 GB | Fits | Q4_K_M | 12K | ~13 tok/s |
| Intel Arc Pro B60 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~11 tok/s |
| RTX 3090 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~28 tok/s |
| Radeon RX 7900 XTX 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~25 tok/s |
| RTX 3090 Ti 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~31 tok/s |
| RTX 4090 24GB | 24.0 GB | Fits | Q4_K_M | 12K | ~31 tok/s |
| Radeon AI PRO R9700 32GB | 32.0 GB | Fits | Q6_K | 32K | ~17 tok/s |
| RTX 5090 32GB | 32.0 GB | Fits | Q6_K | 32K | ~54 tok/s |
| Apple M4 Pro 48GB | 36.0 GB | Fits | Q6_K | 32K | ~8.4 tok/s |
| A100 40GB | 40.0 GB | Fits | Q8_0 | 32K | ~47 tok/s |
| Apple M5 Pro 64GB | 48.0 GB | Fits | Q8_0 | 32K | ~9.5 tok/s |
| Apple M1 Max 64GB | 48.0 GB | Fits | Q8_0 | 32K | ~12 tok/s |
| Radeon RX 7900 XT 20GB | 20.0 GB | 88% on GPU | Q3_K_M | – | ~9.7 tok/s |
| RTX 4060 Ti 16GB | 16.0 GB | 70% on GPU | Q2_K | – | ~4.4 tok/s |
| RTX 4070 Ti Super 16GB | 16.0 GB | 70% on GPU | Q2_K | – | ~5.5 tok/s |
| RTX 4080 16GB | 16.0 GB | 70% on GPU | Q2_K | – | ~5.5 tok/s |
| RTX 4080 Super 16GB | 16.0 GB | 70% on GPU | Q2_K | – | ~5.6 tok/s |
| RTX 5060 Ti 16GB | 16.0 GB | 70% on GPU | Q2_K | – | ~5.0 tok/s |
About Qwen 2.5 Coder 32B
The coding variant of Qwen 2.5 32B, with identical memory needs. Coding agents send long files and repository context, so plan VRAM for 32K tokens rather than 8K: that adds about 8 GB of FP16 cache.
| Spec | Value |
|---|---|
| Parameters | 32.8B |
| Layers | 64 |
| Hidden size | 5120 |
| Attention heads / KV heads | 40 / 8 |
| Head dimension | 128 |
| Vocabulary | 152,064 |
| Max context (used here) | 32K tokens |
| KV cache per token (FP16) | 256 KB |
| Released | 2024-11 |
| License | Apache 2.0 |
Qwen 2.5 Coder 32B VRAM: common questions
What context length should I plan for with a coding model?
At least 32K for editor and agent use. For Qwen 2.5 Coder 32B that means about 27 GB at Q4_K_M with an FP16 cache, about 23.6 GB with a Q8_0 cache (a tight fit on 24 GB), or about 21.6 GB with a Q4_0 cache.
How much VRAM does Qwen 2.5 Coder 32B need?
About 21.4 GB at Q4_K_M with an 8K context, 35.3 GB at Q8_0 and 64.0 GB at FP16. Longer contexts add 0.25 GB per 1,000 tokens with an FP16 cache.
What is the cheapest GPU that can run Qwen 2.5 Coder 32B?
The smallest listed device that fits it fully at Q4_K_M with 8K context is the NVIDIA L4 24GB (24.0 GB usable).