CPU Offloading: Running Models Bigger Than Your VRAM
When a model does not fit in VRAM, runtimes like llama.cpp, Ollama and LM Studio put the remaining layers in system RAM. It works, but system RAM is 5–15× slower than VRAM, so speed drops sharply as soon as a meaningful share is offloaded. MoE models are the exception: they tolerate offloading far better.
How speed falls with offloading
Qwen 3 32B at Q4_K_M needs 21.4 GB. Estimated generation speed on cards that hold different shares of it:
| GPU | Share in VRAM | Speed |
|---|---|---|
| RTX 5090 32GB | 100% | ~54 tok/s |
| RTX 3090 24GB | 100% | ~28 tok/s |
| RTX 4060 Ti 16GB | 70% | ~4.4 tok/s |
| RTX 3060 12GB | 53% | ~3.5 tok/s |
Even with 70% on the GPU, the 30% in system RAM dominates the time per token. Our estimates assume dual-channel DDR5 (about 60 GB/s usable).
Dense vs MoE offloading
Compare the dense 32B above with Qwen 3.6 35B-A3B (21.0 GB) on the same RTX 4060 Ti 16GB: ~40 tokens/s versus ~4.4. The MoE model only reads its active experts from RAM, so offloading costs much less.
What matters for offload speed
- Memory channels: dual-channel DDR5-6000 gives about 96 GB/s peak; quad- or eight-channel workstation and server platforms are several times faster.
- DDR4 vs DDR5: DDR4-3200 dual channel is about half the bandwidth of DDR5-6400.
- CPU cores: matter for prompt processing; generation is mostly bandwidth-bound.
- What you offload: for MoE models offload the experts, not whole layers (
--n-cpu-moein llama.cpp).
When offloading is worth it
- Occasional use of a model one step too big, such as Llama 3.3 70B on a single 24 GB card at ~1.8 tokens/s.
- Large MoE models with expert offload.
- Batch jobs where speed does not matter.
For interactive chat or coding, a smaller model or quantization that fits fully almost always feels better. The Can I Run It tool shows which models fit your GPU fully and which need offloading.
How to offload in each runtime
- llama.cpp:
-ngl Nsets how many layers go to the GPU. For MoE models,--n-cpu-moe Nkeeps the experts of N layers on the CPU while everything else stays on the GPU. - Ollama: offloads automatically;
num_gpuoverrides the number of GPU layers. - LM Studio: the GPU Offload slider.
- vLLM:
--cpu-offload-gbmoves part of the weights to system RAM, but vLLM is built for models that fit in VRAM.
Estimated speed by RAM type
When part of a dense model runs from system RAM, that part's speed is set by memory bandwidth. Rough peak figures:
| System memory | Peak bandwidth | Typical use |
|---|---|---|
| DDR4-3200, dual channel | ~51 GB/s | Older desktops |
| DDR5-6000, dual channel | ~96 GB/s | Current desktops |
| DDR5, quad channel | ~190 GB/s | Threadripper, Xeon W |
| DDR5, 8–12 channels | 300–600 GB/s | EPYC servers |
Server platforms with many memory channels can run large MoE models from RAM alone at usable speed, which is how some people run very large models without large GPUs.
Keep the cache on the GPU
When offloading, keep the KV cache and attention on the GPU if possible. Prompt processing then stays fast, and only the feed-forward work for offloaded layers runs on the CPU. In LM Studio this is the "Offload KV cache to GPU memory" option.
Signs you should not offload
- Interactive chat feels sluggish under about 5 tokens per second.
- Your system starts swapping to disk because RAM is full.
- A smaller model or quantization would fit fully.