What LLMs Can the H100 80GB SXM Run?

The H100 80GB SXM has 80 GB of VRAM and 3350 GB/s of bandwidth. It runs 50 of 58 models fully at Q4_K_M with 8K context, up to Qwen 3.5 122B-A10B, and generates about 357 tokens/s on an 8B model.

NVIDIA · Hopper · Released 2022-09 · Updated October 14, 2026

51 of 58 models fit

H100 80GB SXM (80.0 GB) at Q4_K_M with 8K context. Largest that fits fully: Qwen 3.5 122B-A10B.

ModelNeedsStatusSpeedBest quant that fits
Qwen 3.5 122B-A10B70.2 GBFits~130 tok/sQ4_K_M
Mistral Small 4 119B68.2 GBFits~183 tok/sQ4_K_M
gpt-oss-120b67.2 GBFits~196 tok/sMXFP4 (official)
Llama 4 Scout64.0 GBFits~101 tok/sQ5_K_M
GLM-4.5-Air62.1 GBFits~121 tok/sQ5_K_M
Qwen3-Next 80B-A3B46.3 GBFits~175 tok/sQ6_K
Qwen 2.5 72B44.5 GBFits~46 tok/sQ6_K
Llama 3.3 70B43.2 GBFits~47 tok/sQ8_0
DeepSeek R1 Distill Llama 70B43.2 GBFits~47 tok/sQ8_0
Mixtral 8x7B28.0 GBFits~142 tok/sQ8_0
Qwen 2.5 32B21.4 GBFits~97 tok/sFP16 / BF16
Qwen 2.5 Coder 32B21.4 GBFits~97 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 32B21.4 GBFits~97 tok/sFP16 / BF16
Qwen 3 32B21.4 GBFits~97 tok/sFP16 / BF16
Qwen 3.5 35B-A3B21.0 GBFits~204 tok/sFP16 / BF16
Qwen 3.6 35B-A3B21.0 GBFits~204 tok/sFP16 / BF16
Gemma 4 31B19.8 GBFits~103 tok/sFP16 / BF16
Qwen 3 30B-A3B18.8 GBFits~169 tok/sFP16 / BF16
Qwen3-Coder 30B-A3B18.8 GBFits~169 tok/sFP16 / BF16
Gemma 3 27B17.6 GBFits~116 tok/sFP16 / BF16
Qwen 3.5 27B16.9 GBFits~120 tok/sFP16 / BF16
Qwen 3.6 27B16.9 GBFits~120 tok/sFP16 / BF16
Qwen 3.8 27B16.9 GBFits~120 tok/sFP16 / BF16
Mistral Small 3.1 24B15.6 GBFits~135 tok/sFP16 / BF16
Gemma 4 26B-A4B15.6 GBFits~239 tok/sFP16 / BF16
gpt-oss-20b12.9 GBFits~290 tok/sMXFP4 (official)
Qwen 2.5 14B10.7 GBFits~202 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 14B10.7 GBFits~202 tok/sFP16 / BF16
Phi-4 14B10.6 GBFits~205 tok/sFP16 / BF16
Qwen 3 14B10.5 GBFits~207 tok/sFP16 / BF16
Ministral 3 14B9.7 GBFits~224 tok/sFP16 / BF16
Mistral Nemo 12B9.0 GBFits~244 tok/sFP16 / BF16
Gemma 3 12B8.8 GBFits~241 tok/sFP16 / BF16
Ministral 3 8B6.7 GBFits~337 tok/sFP16 / BF16
Qwen 3 8B6.6 GBFits~343 tok/sFP16 / BF16
Qwen 3.5 9B6.4 GBFits~345 tok/sFP16 / BF16
Llama 3.1 8B6.3 GBFits~357 tok/sFP16 / BF16
DeepSeek R1 Distill Llama 8B6.3 GBFits~357 tok/sFP16 / BF16
Mistral 7B v0.35.7 GBFits~387 tok/sFP16 / BF16
Gemma 4 E4B5.7 GBFits~372 tok/sFP16 / BF16
Qwen 2.5 7B5.6 GBFits~399 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 7B5.6 GBFits~399 tok/sFP16 / BF16
Qwen 3 4B4.2 GBFits~569 tok/sFP16 / BF16
Phi-4 Mini4.1 GBFits~611 tok/sFP16 / BF16
Gemma 4 E2B4.0 GBFits~560 tok/sFP16 / BF16
Gemma 3 4B3.7 GBFits~613 tok/sFP16 / BF16
Qwen 3.5 4B3.7 GBFits~635 tok/sFP16 / BF16
Ministral 3 3B3.5 GBFits~707 tok/sFP16 / BF16
Llama 3.2 3B3.5 GBFits~720 tok/sFP16 / BF16
Llama 3.2 1B1.7 GBFits~1764 tok/sFP16 / BF16
Gemma 3 1B1.6 GBFits~1814 tok/sFP16 / BF16
Llama 3.1 405B234 GB32% on GPU~0.25 tok/s–
Qwen 3.5 397B-A17B226 GB33% on GPU~5.7 tok/s–
DeepSeek V4 Flash161 GB47% on GPU~9.4 tok/s–
Qwen 3 235B-A22B135 GB56% on GPU~6.2 tok/s–
MiniMax-M2133 GB57% on GPU~12 tok/s–
Kimi K2.6566 GBToo small––
DeepSeek R1 671B380 GBToo small––

H100 80GB SXM for local AI

The standard datacenter GPU for LLM serving, with native FP8. Same memory as the A100 80GB but about 1.6 times the bandwidth. PCIe versions have lower bandwidth (2 TB/s).

Among the 2 devices we track with about 80 GB of usable memory, the H100 80GB SXM is the fastest at generating text, because generation speed follows memory bandwidth. Others in the same class: A100 80GB. They all hold the same models; the faster ones simply answer sooner.

Our picks for the H100 80GB SXM

The model we would run first for each job, chosen from our recommendations by what fits fully with an 8K context. "Best quant" is the highest quality that fits; "max context" is at Q4_K_M or the native format.

TaskModelBest quantMax contextSpeed
Best all-rounderQwen 3.5 122B-A10BQ4_K_M219K~130 tok/s
CodingGLM-4.5-AirQ5_K_M80K~121 tok/s
Fastest repliesQwen 3.5 122B-A10BQ4_K_M219K~130 tok/s
Long documents (64K+)Qwen 3.5 122B-A10BQ4_K_M219K~130 tok/s
Images in, text outGemma 4 31BFP16 / BF16256K~103 tok/s
Image generationFLUX.2 [dev]FP8––
SpecValue
Memory80 GB
Memory bandwidth3350 GB/s
ArchitectureHopper
Released2022-09
Llama 3.1 8B Q4_K_M speed~357 tok/s
Largest model that fitsQwen 3.5 122B-A10B

Image and video models on the H100 80GB SXM

Peak VRAM with model swapping (ComfyUI default on smaller cards).

ModelFP8FP8 fitGGUF Q4Q4 fit
Stable Diffusion 1.51.8 GBFits1.7 GBFits
SDXL 1.04.1 GBFits4.1 GBFits
Stable Diffusion 3.5 Medium6.4 GBFits6.4 GBFits
Stable Diffusion 3.5 Large8.6 GBFits6.4 GBFits
FLUX.1 [dev]12.1 GBFits7.3 GBFits
FLUX.1 [schnell]12.1 GBFits7.3 GBFits
Qwen-Image20.4 GBFits12.3 GBFits
Wan 2.2 A14B (T2V / I2V)18.8 GBFits13.2 GBFits
Wan 2.2 TI2V 5B8.0 GBFits6.5 GBFits
HunyuanVideo18.3 GBFits13.1 GBFits
FLUX.1 Kontext [dev]12.3 GBFits7.5 GBFits
FLUX.2 [dev]31.0 GBFits23.6 GBFits
Z-Image Turbo6.8 GBFits4.9 GBFits
Qwen-Image-Edit 250920.5 GBFits12.4 GBFits
LTX-219.2 GBFits12.6 GBFits

Upgrade path

The next step up is the NVIDIA H200 141GB with 141 GB and 43% more bandwidth.

Comparisons

H100 80GB SXM for LLMs: common questions

Can the H100 80GB SXM run Llama 3.3 70B?

Yes. Llama 3.3 70B needs about 43.2 GB at Q4_K_M with 8K context, and the best quality that fits is Q8_0.

Can the H100 80GB SXM run Qwen 3 32B?

Yes. Qwen 3 32B needs about 21.4 GB at Q4_K_M with 8K context, and the best quality that fits is FP16 / BF16.

Can the H100 80GB SXM run Qwen 3 14B?

Yes. Qwen 3 14B needs about 10.5 GB at Q4_K_M with 8K context, and the best quality that fits is FP16 / BF16.

How fast is the H100 80GB SXM for LLMs?

About 357 tokens per second for an 8B model at Q4_K_M, based on its 3350 GB/s memory bandwidth. Larger models scale down roughly in proportion to their size.

Related