Llama 3.1 405B VRAM Requirements
Llama 3.1 405B needs about 234 GB of VRAM at Q4_K_M with an 8K context: 229 GB for weights, 3.9 GB for the KV cache and 1.0 GB of runtime overhead. At Q8_0 it needs 407 GB and at full FP16 precision 761 GB. The smallest GPU that runs it fully is the Apple M3 Ultra 512GB.
234 GB VRAM
Llama 3.1 405B at Q4_K_M with 8K context and the KV cache in FP16.
- Weights 229 GB
- KV cache 3.9 GB
- Runtime overhead 1.0 GB
Which GPUs can run it
- RTX 3060 12GBToo small
- RTX 4060 Ti 16GBToo small
- RTX 3090 24GBToo small
- Radeon RX 7900 XTX 24GBToo small
- RTX 4090 24GBToo small
- RTX 5090 32GBToo small
- RTX 6000 Ada 48GBToo small
- A100 80GB32% on GPU~0.25 tok/s
- H100 80GB SXM32% on GPU~0.25 tok/s
- Ryzen AI Max+ 395 128GBToo small
- Apple M4 Max 128GBToo small
- Apple M5 Max 128GBToo small
- RTX PRO 6000 Blackwell 96GB39% on GPU~0.27 tok/s
- NVIDIA DGX Spark 128GBToo small
- Apple M3 Ultra 512GBFits~2.1 tok/s
Llama 3.1 405B VRAM by quantization
Total memory with an FP16 KV cache. "Smallest tier" is the smallest standard VRAM size with a 6% safety margin at 8K.
| Quantization | Bits/weight | Weights | Total at 8K | Total at 32K | Smallest tier |
|---|---|---|---|---|---|
| Q4_K_M | 4.85 | 229 GB | 234 GB | 246 GB | Multi-GPU / server |
| Q5_K_M | 5.69 | 269 GB | 274 GB | 286 GB | Multi-GPU / server |
| Q6_K | 6.59 | 311 GB | 316 GB | 328 GB | Multi-GPU / server |
| Q8_0 | 8.50 | 402 GB | 407 GB | 418 GB | Multi-GPU / server |
| FP16 / BF16 | 16.00 | 756 GB | 761 GB | 773 GB | Multi-GPU / server |
| Q3_K_M | 3.91 | 185 GB | 190 GB | 202 GB | Multi-GPU / server |
| Q2_K | 3.00 | 142 GB | 147 GB | 159 GB | Multi-GPU / server |
VRAM by context length
At Q4_K_M. The KV cache grows linearly with context: 0.49 GB per 1,000 tokens in FP16.
| Context | KV cache (FP16) | KV cache (Q8_0) | Total, FP16 cache | Total, Q8_0 cache |
|---|---|---|---|---|
| 4K tokens | 2.0 GB | 1.0 GB | 232 GB | 231 GB |
| 8K tokens | 3.9 GB | 2.1 GB | 234 GB | 232 GB |
| 32K tokens | 15.8 GB | 8.4 GB | 246 GB | 239 GB |
| 128K tokens | 63.0 GB | 33.5 GB | 293 GB | 264 GB |
GPUs that can run Llama 3.1 405B
At Q4_K_M with 8K context. "Best quant" is the highest-quality format that fits fully; "max context" is at Q4_K_M.
| GPU | Usable memory | Status | Best quant | Max context | Speed |
|---|---|---|---|---|---|
| Apple M3 Ultra 512GB | 384 GB | Fits | Q6_K | 128K | ~2.1 tok/s |
| AMD Instinct MI300X 192GB | 192 GB | 77% on GPU | Q2_K | – | ~0.70 tok/s |
| NVIDIA H200 141GB | 141 GB | 57% on GPU | – | – | ~0.38 tok/s |
| RTX PRO 6000 Blackwell 96GB | 96.0 GB | 39% on GPU | – | – | ~0.27 tok/s |
| A100 80GB | 80.0 GB | 32% on GPU | – | – | ~0.25 tok/s |
| H100 80GB SXM | 80.0 GB | 32% on GPU | – | – | ~0.25 tok/s |
About Llama 3.1 405B
Meta's largest dense open model. Even at Q4_K_M it needs well over 200 GB, which puts it in multi-GPU server or 512 GB Mac Studio territory. Meta also published an official FP8 version for 8x H100 serving.
| Spec | Value |
|---|---|
| Parameters | 405.9B |
| Layers | 126 |
| Hidden size | 16384 |
| Attention heads / KV heads | 128 / 8 |
| Head dimension | 128 |
| Vocabulary | 128,256 |
| Max context (used here) | 128K tokens |
| KV cache per token (FP16) | 504 KB |
| Released | 2024-07 |
| License | Llama 3.1 Community License |
Llama 3.1 405B VRAM: common questions
Can Llama 3.1 405B run on a Mac Studio?
A 512 GB M3 Ultra can hold Q4_K_M with a moderate context once you raise the GPU memory limit, but expect only a few tokens per second because every token reads about 230 GB of weights.
How much VRAM does Llama 3.1 405B need?
About 234 GB at Q4_K_M with an 8K context, 407 GB at Q8_0 and 761 GB at FP16. Longer contexts add 0.49 GB per 1,000 tokens with an FP16 cache.
What is the cheapest GPU that can run Llama 3.1 405B?
The smallest listed device that fits it fully at Q4_K_M with 8K context is the Apple M3 Ultra 512GB (384 GB usable).