Mixtral 8x7B VRAM Requirements
Mixtral 8x7B needs about 28.0 GB of VRAM at Q4_K_M with an 8K context: 26.4 GB for weights, 1.0 GB for the KV cache and 0.6 GB of runtime overhead. At Q8_0 it needs 47.8 GB and at full FP16 precision 88.6 GB. The smallest GPU that runs it fully is the Radeon AI PRO R9700 32GB.
28.0 GB VRAM
Mixtral 8x7B at Q4_K_M with 8K context and the KV cache in FP16.
- Weights 26.4 GB
- KV cache 1.0 GB
- Runtime overhead 0.6 GB
Mixture of experts: all 46.7B parameters need memory, but only 12.9B are read per token, so it runs at the speed of a much smaller model.
Which GPUs can run it
- RTX 3060 12GB40% on GPU~7.3 tok/s
- RTX 4060 Ti 16GB54% on GPU~8.4 tok/s
- RTX 3090 24GB81% on GPU~19 tok/s
- Radeon RX 7900 XTX 24GB81% on GPU~18 tok/s
- RTX 4090 24GB81% on GPU~19 tok/s
- RTX 5090 32GBFits~96 tok/s
- RTX 6000 Ada 48GBFits~60 tok/s
- A100 80GBFits~105 tok/s
- H100 80GB SXMFits~142 tok/s
- Ryzen AI Max+ 395 128GBFits~17 tok/s
- Apple M4 Max 128GBFits~37 tok/s
- Apple M5 Max 128GBFits~41 tok/s
- RTX PRO 6000 Blackwell 96GBFits~96 tok/s
- NVIDIA DGX Spark 128GBFits~18 tok/s
- Apple M3 Ultra 512GBFits~53 tok/s
Multi-GPU: about 2× 24 GB cards (RTX 3090 / 4090) with the model split across them.
Mixtral 8x7B VRAM by quantization
Total memory with an FP16 KV cache. "Smallest tier" is the smallest standard VRAM size with a 6% safety margin at 8K.
| Quantization | Bits/weight | Weights | Total at 8K | Total at 32K | Smallest tier |
|---|---|---|---|---|---|
| Q4_K_M | 4.85 | 26.4 GB | 28.0 GB | 31.0 GB | 32 GB |
| Q5_K_M | 5.69 | 30.9 GB | 32.6 GB | 35.6 GB | 48 GB |
| Q6_K | 6.59 | 35.8 GB | 37.5 GB | 40.5 GB | 48 GB |
| Q8_0 | 8.50 | 46.2 GB | 47.8 GB | 50.8 GB | 64 GB |
| FP16 / BF16 | 16.00 | 87.0 GB | 88.6 GB | 91.6 GB | 96 GB |
| Q3_K_M | 3.91 | 21.3 GB | 22.9 GB | 25.9 GB | 32 GB |
| Q2_K | 3.00 | 16.3 GB | 17.9 GB | 20.9 GB | 20 GB |
VRAM by context length
At Q4_K_M. The KV cache grows linearly with context: 0.13 GB per 1,000 tokens in FP16.
| Context | KV cache (FP16) | KV cache (Q8_0) | Total, FP16 cache | Total, Q8_0 cache |
|---|---|---|---|---|
| 4K tokens | 0.5 GB | 0.3 GB | 27.5 GB | 27.3 GB |
| 8K tokens | 1.0 GB | 0.5 GB | 28.0 GB | 27.5 GB |
| 32K tokens | 4.0 GB | 2.1 GB | 31.0 GB | 29.1 GB |
GPUs that can run Mixtral 8x7B
At Q4_K_M with 8K context. "Best quant" is the highest-quality format that fits fully; "max context" is at Q4_K_M.
| GPU | Usable memory | Status | Best quant | Max context | Speed |
|---|---|---|---|---|---|
| Radeon AI PRO R9700 32GB | 32.0 GB | Fits | Q4_K_M | 24K | ~37 tok/s |
| RTX 5090 32GB | 32.0 GB | Fits | Q4_K_M | 24K | ~96 tok/s |
| Apple M4 Pro 48GB | 36.0 GB | Fits | Q5_K_M | 32K | ~20 tok/s |
| A100 40GB | 40.0 GB | Fits | Q6_K | 32K | ~87 tok/s |
| Apple M5 Pro 64GB | 48.0 GB | Fits | Q6_K | 32K | ~22 tok/s |
| Apple M1 Max 64GB | 48.0 GB | Fits | Q6_K | 32K | ~28 tok/s |
| Apple M4 Max 64GB | 48.0 GB | Fits | Q6_K | 32K | ~37 tok/s |
| RTX A6000 48GB | 48.0 GB | Fits | Q6_K | 32K | ~50 tok/s |
| L40S 48GB | 48.0 GB | Fits | Q6_K | 32K | ~55 tok/s |
| RTX 6000 Ada 48GB | 48.0 GB | Fits | Q6_K | 32K | ~60 tok/s |
| A100 80GB | 80.0 GB | Fits | Q8_0 | 32K | ~105 tok/s |
| H100 80GB SXM | 80.0 GB | Fits | Q8_0 | 32K | ~142 tok/s |
| Ryzen AI Max+ 395 128GB | 96.0 GB | Fits | FP16 / BF16 | 32K | ~17 tok/s |
| Apple M3 Max 128GB | 96.0 GB | Fits | FP16 / BF16 | 32K | ~28 tok/s |
| RTX 3090 24GB | 24.0 GB | 81% on GPU | Q2_K | – | ~19 tok/s |
| RTX 3090 Ti 24GB | 24.0 GB | 81% on GPU | Q2_K | – | ~19 tok/s |
| RTX 4090 24GB | 24.0 GB | 81% on GPU | Q2_K | – | ~19 tok/s |
| Radeon RX 7900 XTX 24GB | 24.0 GB | 81% on GPU | Q2_K | – | ~18 tok/s |
| Intel Arc Pro B60 24GB | 24.0 GB | 81% on GPU | Q2_K | – | ~14 tok/s |
| NVIDIA L4 24GB | 24.0 GB | 81% on GPU | Q2_K | – | ~13 tok/s |
Multi-GPU: about 2× 24 GB cards (RTX 3090 or 4090) run it at Q4_K_M, using tensor or layer splitting in llama.cpp, vLLM or ExLlamaV2.
About Mixtral 8x7B
The first widely used open mixture-of-experts model: 8 experts, 2 active per token. Memory follows the 46.7B total while speed follows the 12.9B active parameters.
| Spec | Value |
|---|---|
| Parameters | 46.7B total, 12.9B active (MoE) |
| Layers | 32 |
| Hidden size | 4096 |
| Attention heads / KV heads | 32 / 8 |
| Head dimension | 128 |
| Vocabulary | 32,000 |
| Max context (used here) | 32K tokens |
| KV cache per token (FP16) | 128 KB |
| Released | 2023-12 |
| License | Apache 2.0 |
Mixtral 8x7B VRAM: common questions
Why is Mixtral 8x7B 47B and not 56B?
Only the feed-forward layers are duplicated across the 8 experts. Attention and embeddings are shared, so the total is 46.7B.
How much VRAM does Mixtral 8x7B need?
About 28.0 GB at Q4_K_M with an 8K context, 47.8 GB at Q8_0 and 88.6 GB at FP16. Longer contexts add 0.13 GB per 1,000 tokens with an FP16 cache.
What is the cheapest GPU that can run Mixtral 8x7B?
The smallest listed device that fits it fully at Q4_K_M with 8K context is the Radeon AI PRO R9700 32GB (32.0 GB usable). With 24 GB cards you need about 2 of them.
Does Mixtral 8x7B need VRAM for all 46.7B parameters?
Yes. All experts must be loaded because any token can use any expert. Only 12.9B parameters are read per token, so it generates text about as fast as a 13B dense model.