Multi-GPU Setups for Local LLMs

Updated October 14, 2026 · Figures calculated live from our model data

Yes, you can combine VRAM across GPUs for inference: two 24 GB cards hold a model that needs up to about 44 GB. Runtimes split the model's layers between cards. You get the memory of both, but by default not the speed of both; tensor parallelism in vLLM or ExLlama is what makes multiple cards faster.

How many 24 GB GPUs does each model need?

At Q4_K_M (or native format) with an 8K context, allowing for each card's own runtime buffers.

Layer split vs tensor parallel

• Layer split (llama.cpp, Ollama, LM Studio default): card 1 runs the first half of the layers, card 2 the second. Memory adds up; speed stays about the same as one card, because the cards take turns. Works with different GPU models and over slow PCIe links.
• Tensor parallel (vLLM, SGLang, ExLlamaV2/V3, llama.cpp row split): every layer is split across cards, which work at the same time. Generation can be 1.5–1.8× faster with two cards, but it needs matched GPUs and more PCIe bandwidth, and vLLM usually requires the GPU count to divide the number of attention heads (2, 4 or 8 cards).

Commands

# llama.cpp: split by layers, 60/40 between two cards
llama-server -m model.gguf -ngl 99 --split-mode layer --tensor-split 60,40

# vLLM: tensor parallel across two GPUs
vllm serve Qwen/Qwen3-32B-AWQ --tensor-parallel-size 2

Ollama splits automatically across all visible GPUs; limit it with CUDA_VISIBLE_DEVICES.

Mixing different GPUs

With layer split you can pair, say, an RTX 3090 24GB and an RTX 3060 12GB for 36 GB. Set the split in proportion to memory, and expect speed closer to the slower card on its share of layers. Mixing NVIDIA and AMD in one process is possible with llama.cpp's Vulkan backend but not with CUDA or ROCm builds.

Hardware checklist

• PCIe x4 per card is enough for layer-split inference; tensor parallel benefits from x8 or x16.
• NVLink is not needed for inference.
• Budget power for both cards at full load: two 3090s can draw 700 W; power-limiting each to 250–280 W costs little speed for LLMs.
• Check case clearance and cooling; stacked triple-slot cards run hot.

Planning a 70B setup? See running 70B models on consumer GPUs and the 48 GB tier.

Example builds

Motherboards and lanes

Consumer platforms usually have one x16 slot wired to the CPU and a second slot at x4 through the chipset, or an x8/x8 split on higher-end boards. For layer-split inference, x4 is enough because only small activations cross between cards. Workstation platforms (Threadripper, Xeon W) give every card x16 and are worth it for four or more GPUs or for tensor parallelism and training. PCIe risers let you space cards out for cooling; keep them short and use PCIe 4.0-rated cables.

Cooling and power

Two open-air cards stacked next to each other will run the top card hot. Blower-style cards, a gap of one slot, or vertical mounting help. Set a power limit with nvidia-smi -pl 280 on each 3090; LLM speed barely changes and the system becomes far easier to cool and power. Use separate PCIe power cables for each connector rather than daisy-chained ones.

Mixed setups that work well

• A fast card for the display and small models, plus a cheaper high-memory card for overflow.
• A GPU plus a unified-memory machine running separate models over the network, for example a chat model on one and an embedding or coding model on the other.
• Older 24 GB cards like the Tesla P40 24GB as extra memory for llama.cpp only, accepting slower speed on their share of layers.