What LLMs Can the NVIDIA H200 141GB Run?

The NVIDIA H200 141GB has 141 GB of VRAM and 4800 GB/s of bandwidth. It runs 52 of 58 models fully at Q4_K_M with 8K context, up to Qwen 3 235B-A22B, and generates about 487 tokens/s on an 8B model.

NVIDIA · Hopper · Released 2024-11 · Updated October 14, 2026

53 of 58 models fit

NVIDIA H200 141GB (141 GB) at Q4_K_M with 8K context. Largest that fits fully: Qwen 3 235B-A22B.

ModelNeedsStatusSpeedBest quant that fits
Qwen 3.5 122B-A10B70.2 GBFits~147 tok/sQ8_0
Mistral Small 4 119B68.2 GBFits~204 tok/sFP8 (official)
gpt-oss-120b67.2 GBFits~215 tok/sMXFP4 (official)
Llama 4 Scout64.0 GBFits~120 tok/sQ8_0
GLM-4.5-Air62.1 GBFits~139 tok/sQ8_0
Qwen3-Next 80B-A3B46.3 GBFits~184 tok/sQ8_0
Qwen 2.5 72B44.5 GBFits~65 tok/sQ8_0
Llama 3.3 70B43.2 GBFits~67 tok/sQ8_0
DeepSeek R1 Distill Llama 70B43.2 GBFits~67 tok/sQ8_0
Mixtral 8x7B28.0 GBFits~170 tok/sFP16 / BF16
Qwen 2.5 32B21.4 GBFits~136 tok/sFP16 / BF16
Qwen 2.5 Coder 32B21.4 GBFits~136 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 32B21.4 GBFits~136 tok/sFP16 / BF16
Qwen 3 32B21.4 GBFits~136 tok/sFP16 / BF16
Qwen 3.5 35B-A3B21.0 GBFits~216 tok/sFP16 / BF16
Qwen 3.6 35B-A3B21.0 GBFits~216 tok/sFP16 / BF16
Gemma 4 31B19.8 GBFits~144 tok/sFP16 / BF16
Qwen 3 30B-A3B18.8 GBFits~180 tok/sFP16 / BF16
Qwen3-Coder 30B-A3B18.8 GBFits~180 tok/sFP16 / BF16
Gemma 3 27B17.6 GBFits~162 tok/sFP16 / BF16
Qwen 3.5 27B16.9 GBFits~167 tok/sFP16 / BF16
Qwen 3.6 27B16.9 GBFits~167 tok/sFP16 / BF16
Qwen 3.8 27B16.9 GBFits~167 tok/sFP16 / BF16
Mistral Small 3.1 24B15.6 GBFits~189 tok/sFP16 / BF16
Gemma 4 26B-A4B15.6 GBFits~261 tok/sFP16 / BF16
gpt-oss-20b12.9 GBFits~319 tok/sMXFP4 (official)
Qwen 2.5 14B10.7 GBFits~278 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 14B10.7 GBFits~278 tok/sFP16 / BF16
Phi-4 14B10.6 GBFits~284 tok/sFP16 / BF16
Qwen 3 14B10.5 GBFits~286 tok/sFP16 / BF16
Ministral 3 14B9.7 GBFits~309 tok/sFP16 / BF16
Mistral Nemo 12B9.0 GBFits~335 tok/sFP16 / BF16
Gemma 3 12B8.8 GBFits~328 tok/sFP16 / BF16
Ministral 3 8B6.7 GBFits~460 tok/sFP16 / BF16
Qwen 3 8B6.6 GBFits~466 tok/sFP16 / BF16
Qwen 3.5 9B6.4 GBFits~472 tok/sFP16 / BF16
Llama 3.1 8B6.3 GBFits~487 tok/sFP16 / BF16
DeepSeek R1 Distill Llama 8B6.3 GBFits~487 tok/sFP16 / BF16
Mistral 7B v0.35.7 GBFits~526 tok/sFP16 / BF16
Gemma 4 E4B5.7 GBFits~499 tok/sFP16 / BF16
Qwen 2.5 7B5.6 GBFits~545 tok/sFP16 / BF16
DeepSeek R1 Distill Qwen 7B5.6 GBFits~545 tok/sFP16 / BF16
Qwen 3 4B4.2 GBFits~749 tok/sFP16 / BF16
Phi-4 Mini4.1 GBFits~808 tok/sFP16 / BF16
Gemma 4 E2B4.0 GBFits~740 tok/sFP16 / BF16
Gemma 3 4B3.7 GBFits~806 tok/sFP16 / BF16
Qwen 3.5 4B3.7 GBFits~836 tok/sFP16 / BF16
Ministral 3 3B3.5 GBFits~938 tok/sFP16 / BF16
Llama 3.2 3B3.5 GBFits~948 tok/sFP16 / BF16
Llama 3.2 1B1.7 GBFits~2252 tok/sFP16 / BF16
Gemma 3 1B1.6 GBFits~2158 tok/sFP16 / BF16
Qwen 3 235B-A22B135 GBTight fit~72 tok/sIQ4_XS
MiniMax-M2133 GBTight fit~118 tok/sIQ4_XS
DeepSeek R1 671B380 GB35% on GPU~2.8 tok/s–
Llama 3.1 405B234 GB57% on GPU~0.38 tok/s–
Qwen 3.5 397B-A17B226 GB59% on GPU~8.9 tok/s–
DeepSeek V4 Flash161 GB82% on GPU~25 tok/sQ3_K_M
Kimi K2.6566 GBToo small––

NVIDIA H200 141GB for local AI

Hopper with 141 GB of HBM3e. One card holds 70B models in FP8 with a long context or 120B-class MoE models comfortably; eight make the standard node for serving DeepSeek and Kimi-class models.

Our picks for the NVIDIA H200 141GB

The model we would run first for each job, chosen from our recommendations by what fits fully with an 8K context. "Best quant" is the highest quality that fits; "max context" is at Q4_K_M or the native format.

TaskModelBest quantMax contextSpeed
Best all-rounderQwen 3.5 122B-A10BQ8_0256K~147 tok/s
CodingGLM-4.5-AirQ8_0128K~139 tok/s
Fastest repliesQwen 3.5 122B-A10BQ8_0256K~147 tok/s
Long documents (64K+)Qwen 3.5 122B-A10BQ8_0256K~147 tok/s
Images in, text outGemma 4 31BFP16 / BF16256K~144 tok/s
Image generationFLUX.2 [dev]FP8––
SpecValue
Memory141 GB
Memory bandwidth4800 GB/s
ArchitectureHopper
Released2024-11
Llama 3.1 8B Q4_K_M speed~487 tok/s
Largest model that fitsQwen 3 235B-A22B

Image and video models on the NVIDIA H200 141GB

Peak VRAM with model swapping (ComfyUI default on smaller cards).

ModelFP8FP8 fitGGUF Q4Q4 fit
Stable Diffusion 1.51.8 GBFits1.7 GBFits
SDXL 1.04.1 GBFits4.1 GBFits
Stable Diffusion 3.5 Medium6.4 GBFits6.4 GBFits
Stable Diffusion 3.5 Large8.6 GBFits6.4 GBFits
FLUX.1 [dev]12.1 GBFits7.3 GBFits
FLUX.1 [schnell]12.1 GBFits7.3 GBFits
Qwen-Image20.4 GBFits12.3 GBFits
Wan 2.2 A14B (T2V / I2V)18.8 GBFits13.2 GBFits
Wan 2.2 TI2V 5B8.0 GBFits6.5 GBFits
HunyuanVideo18.3 GBFits13.1 GBFits
FLUX.1 Kontext [dev]12.3 GBFits7.5 GBFits
FLUX.2 [dev]31.0 GBFits23.6 GBFits
Z-Image Turbo6.8 GBFits4.9 GBFits
Qwen-Image-Edit 250920.5 GBFits12.4 GBFits
LTX-219.2 GBFits12.6 GBFits

Upgrade path

The next step up is the AMD Instinct MI300X 192GB with 192 GB and 10% more bandwidth.

Comparisons

NVIDIA H200 141GB for LLMs: common questions

Can the NVIDIA H200 141GB run Llama 3.3 70B?

Yes. Llama 3.3 70B needs about 43.2 GB at Q4_K_M with 8K context, and the best quality that fits is Q8_0.

Can the NVIDIA H200 141GB run Qwen 3 32B?

Yes. Qwen 3 32B needs about 21.4 GB at Q4_K_M with 8K context, and the best quality that fits is FP16 / BF16.

Can the NVIDIA H200 141GB run Qwen 3 14B?

Yes. Qwen 3 14B needs about 10.5 GB at Q4_K_M with 8K context, and the best quality that fits is FP16 / BF16.

How fast is the NVIDIA H200 141GB for LLMs?

About 487 tokens per second for an 8B model at Q4_K_M, based on its 4800 GB/s memory bandwidth. Larger models scale down roughly in proportion to their size.

Related