What LLMs Can the A100 40GB Run?

The A100 40GB has 40 GB of VRAM and 1555 GB/s of bandwidth. It runs 42 of 58 models fully at Q4_K_M with 8K context, up to Mixtral 8x7B, and generates about 176 tokens/s on an 8B model.

NVIDIA · Ampere · Released 2020-05 · Updated October 14, 2026

42 of 58 models fit

A100 40GB (40.0 GB) at Q4_K_M with 8K context. Largest that fits fully: Mixtral 8x7B.

ModelNeedsStatusSpeedBest quant that fits
Mixtral 8x7B28.0 GBFits~87 tok/sQ6_K
Qwen 2.5 32B21.4 GBFits~47 tok/sQ8_0
Qwen 2.5 Coder 32B21.4 GBFits~47 tok/sQ8_0
DeepSeek R1 Distill Qwen 32B21.4 GBFits~47 tok/sQ8_0
Qwen 3 32B21.4 GBFits~47 tok/sQ8_0
Qwen 3.5 35B-A3B21.0 GBFits~168 tok/sQ8_0
Qwen 3.6 35B-A3B21.0 GBFits~168 tok/sQ8_0
Gemma 4 31B19.8 GBFits~50 tok/sQ8_0
Qwen 3 30B-A3B18.8 GBFits~139 tok/sQ8_0
Qwen3-Coder 30B-A3B18.8 GBFits~139 tok/sQ8_0
Gemma 3 27B17.6 GBFits~56 tok/sQ8_0
Qwen 3.5 27B16.9 GBFits~58 tok/sQ8_0
Qwen 3.6 27B16.9 GBFits~58 tok/sQ8_0
Qwen 3.8 27B16.9 GBFits~58 tok/sQ8_0
Mistral Small 3.1 24B15.6 GBFits~65 tok/sQ8_0
Gemma 4 26B-A4B15.6 GBFits~180 tok/sQ8_0
gpt-oss-20b12.9 GBFits~215 tok/sMXFP4 (official)
Qwen 2.5 14B10.7 GBFits~99 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 14B10.7 GBFits~99 tok/sFP16 / BF16
Phi-4 14B10.6 GBFits~100 tok/sFP16 / BF16
Qwen 3 14B10.5 GBFits~101 tok/sFP16 / BF16
Ministral 3 14B9.7 GBFits~109 tok/sFP16 / BF16
Mistral Nemo 12B9.0 GBFits~119 tok/sFP16 / BF16
Gemma 3 12B8.8 GBFits~119 tok/sFP16 / BF16
Ministral 3 8B6.7 GBFits~167 tok/sFP16 / BF16
Qwen 3 8B6.6 GBFits~170 tok/sFP16 / BF16
Qwen 3.5 9B6.4 GBFits~170 tok/sFP16 / BF16
Llama 3.1 8B6.3 GBFits~176 tok/sFP16 / BF16
DeepSeek R1 Distill Llama 8B6.3 GBFits~176 tok/sFP16 / BF16
Mistral 7B v0.35.7 GBFits~192 tok/sFP16 / BF16
Gemma 4 E4B5.7 GBFits~188 tok/sFP16 / BF16
Qwen 2.5 7B5.6 GBFits~197 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 7B5.6 GBFits~197 tok/sFP16 / BF16
Qwen 3 4B4.2 GBFits~297 tok/sFP16 / BF16
Phi-4 Mini4.1 GBFits~317 tok/sFP16 / BF16
Gemma 4 E2B4.0 GBFits~291 tok/sFP16 / BF16
Gemma 3 4B3.7 GBFits~320 tok/sFP16 / BF16
Qwen 3.5 4B3.7 GBFits~331 tok/sFP16 / BF16
Ministral 3 3B3.5 GBFits~364 tok/sFP16 / BF16
Llama 3.2 3B3.5 GBFits~375 tok/sFP16 / BF16
Llama 3.2 1B1.7 GBFits~965 tok/sFP16 / BF16
Gemma 3 1B1.6 GBFits~1126 tok/sFP16 / BF16
Qwen 3.5 122B-A10B70.2 GB54% on GPU~13 tok/s–
Mistral Small 4 119B68.2 GB55% on GPU~20 tok/s–
gpt-oss-120b67.2 GB56% on GPU~25 tok/s–
Llama 4 Scout64.0 GB59% on GPU~8.3 tok/s–
GLM-4.5-Air62.1 GB61% on GPU~12 tok/s–
Qwen3-Next 80B-A3B46.3 GB81% on GPU~65 tok/sQ3_K_M
Qwen 2.5 72B44.5 GB85% on GPU~4.9 tok/sQ3_K_M
Llama 3.3 70B43.2 GB87% on GPU~5.7 tok/sQ3_K_M
DeepSeek R1 Distill Llama 70B43.2 GB87% on GPU~5.7 tok/sQ3_K_M
Kimi K2.6566 GBToo small––
DeepSeek R1 671B380 GBToo small––
Llama 3.1 405B234 GBToo small––
Qwen 3.5 397B-A17B226 GBToo small––
DeepSeek V4 Flash161 GBToo small––
Qwen 3 235B-A22B135 GBToo small––
MiniMax-M2133 GBToo small––

A100 40GB for local AI

The original A100. HBM2 bandwidth makes it fast, but 40 GB is too small for 70B models at Q4_K_M.

Our picks for the A100 40GB

The model we would run first for each job, chosen from our recommendations by what fits fully with an 8K context. "Best quant" is the highest quality that fits; "max context" is at Q4_K_M or the native format.

TaskModelBest quantMax contextSpeed
Best all-rounderQwen 3.6 27BQ8_0256K~58 tok/s
CodingQwen 3.6 27BQ8_0256K~58 tok/s
Fastest repliesQwen 3.6 35B-A3BQ8_0256K~168 tok/s
Long documents (64K+)Qwen 3.8 27BQ8_0256K~58 tok/s
Images in, text outGemma 4 31BQ8_0235K~50 tok/s
Image generationFLUX.2 [dev]FP8––
SpecValue
Memory40 GB
Memory bandwidth1555 GB/s
ArchitectureAmpere
Released2020-05
Llama 3.1 8B Q4_K_M speed~176 tok/s
Largest model that fitsMixtral 8x7B

Image and video models on the A100 40GB

Peak VRAM with model swapping (ComfyUI default on smaller cards).

ModelFP8FP8 fitGGUF Q4Q4 fit
Stable Diffusion 1.51.8 GBFits1.7 GBFits
SDXL 1.04.1 GBFits4.1 GBFits
Stable Diffusion 3.5 Medium6.4 GBFits6.4 GBFits
Stable Diffusion 3.5 Large8.6 GBFits6.4 GBFits
FLUX.1 [dev]12.1 GBFits7.3 GBFits
FLUX.1 [schnell]12.1 GBFits7.3 GBFits
Qwen-Image20.4 GBFits12.3 GBFits
Wan 2.2 A14B (T2V / I2V)18.8 GBFits13.2 GBFits
Wan 2.2 TI2V 5B8.0 GBFits6.5 GBFits
HunyuanVideo18.3 GBFits13.1 GBFits
FLUX.1 Kontext [dev]12.3 GBFits7.5 GBFits
FLUX.2 [dev]31.0 GBFits23.6 GBFits
Z-Image Turbo6.8 GBFits4.9 GBFits
Qwen-Image-Edit 250920.5 GBFits12.4 GBFits
LTX-219.2 GBFits12.6 GBFits

Upgrade path

The next step up is the L40S 48GB with 48 GB.

A100 40GB for LLMs: common questions

Can the A100 40GB run Llama 3.3 70B?

Only partly. It needs about 43.2 GB at Q4_K_M, so about 87% fits in VRAM and the rest runs from system RAM, which is slow. Q3_K_M fits fully.

Can the A100 40GB run Qwen 3 32B?

Yes. Qwen 3 32B needs about 21.4 GB at Q4_K_M with 8K context, and the best quality that fits is Q8_0.

Can the A100 40GB run Qwen 3 14B?

Yes. Qwen 3 14B needs about 10.5 GB at Q4_K_M with 8K context, and the best quality that fits is FP16 / BF16.

How fast is the A100 40GB for LLMs?

About 176 tokens per second for an 8B model at Q4_K_M, based on its 1555 GB/s memory bandwidth. Larger models scale down roughly in proportion to their size.

Related