Mistral 7B v0.3 VRAM Requirements
Mistral 7B v0.3 needs about 5.7 GB of VRAM at Q4_K_M with an 8K context: 4.1 GB for weights, 1.0 GB for the KV cache and 0.6 GB of runtime overhead. At Q8_0 it needs 8.8 GB and at full FP16 precision 15.1 GB. The smallest GPU that runs it fully is the RTX 3050 8GB.
5.7 GB VRAM
Mistral 7B v0.3 at Q4_K_M with 8K context and the KV cache in FP16.
- Weights 4.1 GB
- KV cache 1.0 GB
- Runtime overhead 0.6 GB
Which GPUs can run it
- RTX 3060 12GBFits~47 tok/s
- RTX 4060 Ti 16GBFits~37 tok/s
- RTX 3090 24GBFits~119 tok/s
- Radeon RX 7900 XTX 24GBFits~104 tok/s
- RTX 4090 24GBFits~127 tok/s
- RTX 5090 32GBFits~220 tok/s
- RTX 6000 Ada 48GBFits~122 tok/s
- A100 80GBFits~247 tok/s
- H100 80GB SXMFits~387 tok/s
- Ryzen AI Max+ 395 128GBFits~31 tok/s
- Apple M4 Max 128GBFits~70 tok/s
- Apple M5 Max 128GBFits~79 tok/s
- RTX PRO 6000 Blackwell 96GBFits~220 tok/s
- NVIDIA DGX Spark 128GBFits~33 tok/s
- Apple M3 Ultra 512GBFits~104 tok/s
Mistral 7B v0.3 VRAM by quantization
Total memory with an FP16 KV cache. "Smallest tier" is the smallest standard VRAM size with a 6% safety margin at 8K.
| Quantization | Bits/weight | Weights | Total at 8K | Total at 32K | Smallest tier |
|---|---|---|---|---|---|
| Q4_K_M | 4.85 | 4.1 GB | 5.7 GB | 8.7 GB | 8 GB |
| Q5_K_M | 5.69 | 4.8 GB | 6.4 GB | 9.4 GB | 8 GB |
| Q6_K | 6.59 | 5.6 GB | 7.2 GB | 10.2 GB | 8 GB |
| Q8_0 | 8.50 | 7.2 GB | 8.8 GB | 11.8 GB | 10 GB |
| FP16 / BF16 | 16.00 | 13.5 GB | 15.1 GB | 18.1 GB | 20 GB |
| Q3_K_M | 3.91 | 3.3 GB | 4.9 GB | 7.9 GB | 6 GB |
| Q2_K | 3.00 | 2.5 GB | 4.2 GB | 7.2 GB | 6 GB |
VRAM by context length
At Q4_K_M. The KV cache grows linearly with context: 0.13 GB per 1,000 tokens in FP16.
| Context | KV cache (FP16) | KV cache (Q8_0) | Total, FP16 cache | Total, Q8_0 cache |
|---|---|---|---|---|
| 4K tokens | 0.5 GB | 0.3 GB | 5.2 GB | 5.0 GB |
| 8K tokens | 1.0 GB | 0.5 GB | 5.7 GB | 5.2 GB |
| 32K tokens | 4.0 GB | 2.1 GB | 8.7 GB | 6.8 GB |
GPUs that can run Mistral 7B v0.3
At Q4_K_M with 8K context. "Best quant" is the highest-quality format that fits fully; "max context" is at Q4_K_M.
| GPU | Usable memory | Status | Best quant | Max context | Speed |
|---|---|---|---|---|---|
| RTX 3050 8GB | 8.0 GB | Fits | Q6_K | 22K | ~29 tok/s |
| RTX 4060 8GB | 8.0 GB | Fits | Q6_K | 22K | ~35 tok/s |
| RTX 4060 Ti 8GB | 8.0 GB | Fits | Q6_K | 22K | ~37 tok/s |
| Radeon RX 7600 8GB | 8.0 GB | Fits | Q6_K | 22K | ~32 tok/s |
| RTX 3060 Ti 8GB | 8.0 GB | Fits | Q6_K | 22K | ~58 tok/s |
| RTX 3070 8GB | 8.0 GB | Fits | Q6_K | 22K | ~58 tok/s |
| RTX 5060 8GB | 8.0 GB | Fits | Q6_K | 22K | ~58 tok/s |
| RTX 5060 Ti 8GB | 8.0 GB | Fits | Q6_K | 22K | ~58 tok/s |
| RTX 3070 Ti 8GB | 8.0 GB | Fits | Q6_K | 22K | ~78 tok/s |
| Intel Arc B570 10GB | 10.0 GB | Fits | Q8_0 | 32K | ~38 tok/s |
| RTX 3080 10GB | 10.0 GB | Fits | Q8_0 | 32K | ~97 tok/s |
| Apple M4 16GB | 10.7 GB | Fits | Q8_0 | 32K | ~16 tok/s |
| RTX 2080 Ti 11GB | 11.0 GB | Fits | Q8_0 | 32K | ~79 tok/s |
| RTX 2060 12GB | 12.0 GB | Fits | Q8_0 | 32K | ~44 tok/s |
About Mistral 7B v0.3
The model that popularized grouped-query attention in open 7B models. Its small 32K vocabulary means less memory goes to embeddings, so its weights are lighter than Llama 3.1 8B's at every quantization.
| Spec | Value |
|---|---|
| Parameters | 7.25B |
| Layers | 32 |
| Hidden size | 4096 |
| Attention heads / KV heads | 32 / 8 |
| Head dimension | 128 |
| Vocabulary | 32,768 |
| Max context (used here) | 32K tokens |
| KV cache per token (FP16) | 128 KB |
| Released | 2024-05 |
| License | Apache 2.0 |
Mistral 7B v0.3 VRAM: common questions
Is Mistral 7B still worth running?
It remains a common base for fine-tunes and is very light, but newer models of the same size usually score higher. Memory needs are nearly identical to Llama 3.1 8B.
How much VRAM does Mistral 7B v0.3 need?
About 5.7 GB at Q4_K_M with an 8K context, 8.8 GB at Q8_0 and 15.1 GB at FP16. Longer contexts add 0.13 GB per 1,000 tokens with an FP16 cache.
What is the cheapest GPU that can run Mistral 7B v0.3?
The smallest listed device that fits it fully at Q4_K_M with 8K context is the RTX 3050 8GB (8.0 GB usable).