Mixture-of-Experts Models and VRAM

Updated October 14, 2026 · Figures calculated live from our model data

A mixture-of-experts (MoE) model needs memory for all of its parameters, but reads only the active ones for each token. gpt-oss-120b has 116.8B parameters and needs 62.0 GB, but only 5.1B are used per token, so it generates text about as fast as a 5B dense model.

Memory and speed of popular MoE models

Q4_K_M or native format with an 8K context. Speeds are single-user estimates.

On the RTX 3090, models larger than 24 GB run with partial CPU offload, which is why their speed drops.

Why all experts must be in memory

A router picks a few experts for each token in each layer, and the next token may pick completely different ones. Loading experts from disk on demand would be far too slow, so every expert stays in memory. The "A3B" or "A10B" in a model name tells you the active parameters, not the memory you need.

Why MoE suits big-memory, lower-bandwidth hardware

Generation speed depends on bytes read per token. A dense 70B model reads about 39.9 GB per token at Q4_K_M; gpt-oss-120b reads about 2.7 GB. On a Ryzen AI Max+ 395 128GB or Mac, that is the difference between single-digit and 40+ tokens per second.

Expert offloading: big MoE on a small GPU

Because each token touches only a few experts, MoE models tolerate CPU offloading better than dense ones, especially if you put only the experts in system RAM and keep attention and shared layers on the GPU:

# llama.cpp: keep the expert tensors of the first 30 layers on the CPU
llama-server -m glm-4.5-air-Q4_K_M.gguf -ngl 99 --n-cpu-moe 30

# finer control with tensor overrides
llama-server -m model.gguf -ngl 99 -ot ".ffn_.*_exps.=CPU"

With 64 GB of system RAM and a 24 GB GPU, this runs GLM-4.5-Air or gpt-oss-120b at usable speed. See CPU offloading for the general case.

Dense or MoE: which should you run?

• Plenty of memory, modest bandwidth (Mac, Strix Halo, DGX Spark): MoE.
• One fast GPU with limited VRAM: a dense model that fits, or a small-active MoE such as Qwen 3.6 35B-A3B.
• Fine-tuning: dense models are simpler and better supported.

How routing works

In a dense model, every token passes through every feed-forward block. In a mixture-of-experts model, each layer has many smaller feed-forward blocks (experts) and a small router that picks a few of them for each token. Qwen 3.6 35B-A3B has 256 experts per layer and uses 8 routed experts plus one shared expert per token. Attention layers, embeddings and shared experts are used by every token; only the routed experts vary.

Why MoE models are not as fast as their active size suggests

A model with 3B active parameters would ideally run as fast as a 3B dense model. In practice it runs slower, because the GPU launches many small expert computations per layer and the router adds work. Our speed model includes this overhead, which is why our MoE speeds are lower than a simple bandwidth calculation suggests and closer to what runtimes actually report.

Quantizing MoE models

Experts hold most of the parameters, so they dominate file size. Some quantizers keep attention and shared layers at higher precision and quantize experts more aggressively, which saves memory with little quality loss. gpt-oss-120b ships with its experts in MXFP4 and everything else in higher precision, which is why its native file is only 60.8 GB.

Memory planning for MoE on a GPU plus RAM

Add your operating system's needs on top of the RAM figure.