Can I run a 70B model on an RTX 4090?

Not fully. At Q4_K_M it needs about 43.2 GB. Even Q2_K needs about 28.0 GB, just over 24 GB, so a 4090 needs CPU offload.

Do I need NVLink for two GPUs?

No. llama.cpp, Ollama and vLLM split models over PCIe. NVLink helps tensor-parallel training more than single-user inference.

Is a 70B model much better than a 32B model?

Less than it used to be. Recent 27–35B models close much of the gap on many tasks. A 70B model still has more knowledge and handles nuance better, but needs about twice the memory.

Can I run a 70B model on a 64GB Mac?

Yes at Q4_K_M with a short context, since it needs about 43.2 GB and a 64 GB Mac gives the GPU about 48 GB by default. Expect around ~8.1 tokens per second on an M4 Max.

How to Run a 70B Model on Consumer GPUs

Updated October 14, 2026 · Figures calculated live from our model data

A 70B model needs about 43.2 GB at Q4_K_M with 8K context, so it does not fit on any single consumer GPU. The practical options are two 24 GB cards, one 48 GB card, a 64 GB+ Mac or Strix Halo system, or heavy quantization with offloading.

The numbers

Setup	Quant	VRAM needed	Speed (est.)
2× RTX 3090 24GB	Q4_K_M	43.2 GB of 48 GB	~14 tok/s
1× RTX 6000 Ada 48GB	Q4_K_M	43.2 GB	~14 tok/s
Apple M4 Max 64GB	Q4_K_M	42.0 GB (4K ctx)	~8.1 tok/s
Apple M4 Max 128GB	Q8_0	73.2 GB	~4.6 tok/s
Ryzen AI Max+ 395 128GB	Q4_K_M	43.2 GB	~3.5 tok/s
1× RTX 5090 32GB	Q2_K	28.0 GB	~41 tok/s
1× RTX 4090 24GB + RAM offload	Q4_K_M	43.2 GB	~1.9 tok/s

Speeds are bandwidth-based estimates for single-user generation. Layer-split multi-GPU runs the cards one after another, so two 3090s generate at roughly single-card speed on the full model.

Option 1: two 24 GB GPUs

The most popular route. llama.cpp and Ollama split layers across cards automatically; vLLM and ExLlamaV2 support tensor parallelism for more speed. Two used 3090s give 48 GB, enough for Q4_K_M with about 12K–14K of context, or more with a Q8_0 KV cache.

Option 2: one 48 GB workstation card

An RTX A6000 or RTX 6000 Ada does the same job in one slot at around 300 W. It is simpler and quieter, but usually more expensive than two used 3090s.

Option 3: unified memory

A Mac with 64 GB runs Q4_K_M with a short context; 96–128 GB runs Q6_K or Q8_0 comfortably. AMD Strix Halo and NVIDIA DGX Spark systems hold it too. These are quiet and power-efficient but generate at single-digit tokens per second on dense 70B models.

Option 4: squeeze it onto one consumer card

Q2_K needs about 28.0 GB, which fits a 32 GB RTX 5090 32GB but not a 24 GB card. Quality loss at this level is noticeable, and a strong 32B model at Q4_K_M (Qwen 3 32B) is often the better choice.

Option 5: offload to system RAM

With a 24 GB card and 64 GB of RAM, llama.cpp can run Q4_K_M with about half the layers on the CPU. It works for occasional use, but expect only a couple of tokens per second.

Check other 70B-class models: Qwen 2.5 72B, DeepSeek R1 Distill Llama 70B. Or consider MoE alternatives like gpt-oss-120b, which run much faster on unified memory.

Context length on a 48 GB setup

Two 24 GB cards leave limited room for context once 39.9 GB of Q4_K_M weights are loaded. How far you can go depends on the cache type:

Context	FP16 cache	Q8_0 cache	Fits 2× 24 GB (FP16 cache)?
4K	42.0 GB	41.4 GB	Yes
8K	43.2 GB	42.1 GB	Yes
16K	45.7 GB	43.4 GB	No
32K	50.7 GB	46.0 GB	No

With a Q8_0 cache, 2× 24 GB can hold about 16K of context, which is enough for most chat and document work. For more, add a third card or use a 48 GB workstation card.

Which 70B-class model?

Llama 3.3 70B: a strong general model with a large ecosystem of fine-tunes.
Qwen 2.5 72B: slightly larger, strong at multilingual and maths tasks.
DeepSeek R1 Distill Llama 70B: a reasoning model that writes long chains of thought, so it needs more context.
Consider whether you need a dense 70B at all. Qwen3-Next 80B-A3B and gpt-oss-120b are mixture-of-experts models that are far faster on the same memory.

Step by step: two GPUs with llama.cpp

# both cards visible, all layers on GPU, split by layer
llama-server -m Llama-3.3-70B-Instruct-Q4_K_M.gguf \
  -ngl 99 --split-mode layer -c 16384 -fa on -ctk q8_0 -ctv q8_0

Ollama does the same automatically when it sees two GPUs. Check with ollama ps that the model shows 100% GPU. If one card also drives your display, give it a slightly smaller share with --tensor-split 45,55.

Troubleshooting

Out of memory on the first long prompt: lower the context or the batch size (-b 512); prompt processing needs temporary buffers.
Very slow speed: some layers are on the CPU. Check the load log for the number of offloaded layers.
Second GPU not used: check nvidia-smi shows both cards and that CUDA_VISIBLE_DEVICES is not set to one of them.

More on splitting models: multi-GPU setups.