Mistral Small 3.1 24B VRAM Requirements

Mistral Small 3.1 24B needs about 15.6 GB of VRAM at Q4_K_M with an 8K context: 13.6 GB for weights, 1.3 GB for the KV cache and 0.8 GB of runtime overhead. At Q8_0 it needs 25.8 GB and at full FP16 precision 46.8 GB. The smallest GPU that runs it fully is the Apple M4 Pro 24GB.

By Mistral AI · Updated October 14, 2026 · Values from config.json on Hugging Face

15.6 GB VRAM

Mistral Small 3.1 24B at Q4_K_M with 8K context and the KV cache in FP16.

  • Weights 13.6 GB
  • KV cache 1.3 GB
  • Runtime overhead 0.8 GB

Which GPUs can run it

Mistral Small 3.1 24B VRAM by quantization

Total memory with an FP16 KV cache. "Smallest tier" is the smallest standard VRAM size with a 6% safety margin at 8K.

QuantizationBits/weightWeightsTotal at 8KTotal at 32KSmallest tier
Q4_K_M4.8513.6 GB15.6 GB19.4 GB20 GB
Q5_K_M5.6915.9 GB18.0 GB21.7 GB20 GB
Q6_K6.5918.4 GB20.5 GB24.2 GB24 GB
Q8_08.5023.7 GB25.8 GB29.6 GB32 GB
FP16 / BF1616.0044.7 GB46.8 GB50.5 GB64 GB
Q3_K_M3.9110.9 GB13.0 GB16.8 GB16 GB
Q2_K3.008.4 GB10.5 GB14.2 GB12 GB

VRAM by context length

At Q4_K_M. The KV cache grows linearly with context: 0.16 GB per 1,000 tokens in FP16.

ContextKV cache (FP16)KV cache (Q8_0)Total, FP16 cacheTotal, Q8_0 cache
4K tokens0.6 GB0.3 GB15.0 GB14.7 GB
8K tokens1.3 GB0.7 GB15.6 GB15.0 GB
32K tokens5.0 GB2.7 GB19.4 GB17.0 GB
128K tokens20.0 GB10.6 GB34.4 GB25.0 GB

GPUs that can run Mistral Small 3.1 24B

At Q4_K_M with 8K context. "Best quant" is the highest-quality format that fits fully; "max context" is at Q4_K_M.

GPUUsable memoryStatusBest quantMax contextSpeed
Apple M4 Pro 24GB16.0 GBTight fitIQ4_XS4K~12 tok/s
RTX 4060 Ti 16GB16.0 GBTight fitIQ4_XS4K~12 tok/s
Radeon RX 9060 XT 16GB16.0 GBTight fitIQ4_XS4K~12 tok/s
Tesla T4 16GB16.0 GBTight fitIQ4_XS4K~14 tok/s
RTX 5060 Ti 16GB16.0 GBTight fitIQ4_XS4K~19 tok/s
Radeon RX 6800 XT 16GB16.0 GBTight fitIQ4_XS4K~18 tok/s
Intel Arc A770 16GB16.0 GBTight fitIQ4_XS4K~18 tok/s
Radeon RX 7800 XT 16GB16.0 GBTight fitIQ4_XS4K~22 tok/s
Radeon RX 9070 XT 16GB16.0 GBTight fitIQ4_XS4K~23 tok/s
RTX 4070 Ti Super 16GB16.0 GBTight fitIQ4_XS4K~28 tok/s
RTX 4080 16GB16.0 GBTight fitIQ4_XS4K~30 tok/s
RTX 4080 Super 16GB16.0 GBTight fitIQ4_XS4K~31 tok/s
RTX 5070 Ti 16GB16.0 GBTight fitIQ4_XS4K~38 tok/s
RTX 5080 16GB16.0 GBTight fitIQ4_XS4K~40 tok/s
RTX 2060 12GB12.0 GB72% on GPUQ2_K–~6.6 tok/s
RTX 3060 12GB12.0 GB72% on GPUQ2_K–~6.7 tok/s
RTX 3080 12GB12.0 GB72% on GPUQ2_K–~8.3 tok/s
RTX 4070 12GB12.0 GB72% on GPUQ2_K–~7.4 tok/s
RTX 4070 Super 12GB12.0 GB72% on GPUQ2_K–~7.4 tok/s
RTX 4070 Ti 12GB12.0 GB72% on GPUQ2_K–~7.4 tok/s

About Mistral Small 3.1 24B

A dense 24B model with vision input, designed to run on a single RTX 4090 or a 32 GB Mac. It has only 40 layers for its size, so its KV cache is modest: 160 KB per token in FP16.

SpecValue
Parameters24B
Layers40
Hidden size5120
Attention heads / KV heads32 / 8
Head dimension128
Vocabulary131,072
Max context (used here)128K tokens
KV cache per token (FP16)160 KB
Released2025-03
LicenseApache 2.0

Mistral Small 3.1 24B VRAM: common questions

Does Mistral Small 3.1 fit on a 16GB GPU?

At Q4_K_M the weights are about 13.6 GB, so a 16 GB card fits it only with a short context (about 3K tokens). 24 GB is comfortable.

How much VRAM does Mistral Small 3.1 24B need?

About 15.6 GB at Q4_K_M with an 8K context, 25.8 GB at Q8_0 and 46.8 GB at FP16. Longer contexts add 0.16 GB per 1,000 tokens with an FP16 cache.

What is the cheapest GPU that can run Mistral Small 3.1 24B?

The smallest listed device that fits it fully at Q4_K_M with 8K context is the Apple M4 Pro 24GB (16.0 GB usable).

Related