Best GPU for Local LLMs
For local LLMs, buy memory first and bandwidth second. VRAM decides which models you can run; bandwidth decides how fast. A used RTX 3090 24GB remains the best value for 24 GB, and the RTX 3060 12GB is the best entry card.
Why memory matters more than compute
Generating text is limited by how fast the GPU can read weights from memory, not by its compute power. That is why a five-year-old RTX 3090 24GB generates about 109 tokens/s on an 8B model while an RTX 4090 24GB manages about 117. Compute matters more for prompt processing, image generation and training.
Picks by goal
| Goal | Pick | Why |
|---|---|---|
| Cheapest useful card | RTX 3060 12GB | 12 GB runs 14B models; low used prices |
| Best new budget 16 GB | RTX 5060 Ti 16GB | 16 GB with 448 GB/s; runs 24B models at Q4 |
| Best value for 32B models | RTX 3090 24GB (used) | 24 GB at the lowest price; two make 48 GB for 70B |
| Fastest single consumer card | RTX 5090 32GB | 32 GB and 1792 GB/s |
| AMD alternative | Radeon RX 7900 XTX 24GB | 24 GB; great with llama.cpp, weaker for training |
| 70B on one card | RTX 6000 Ada 48GB / RTX A6000 48GB | 48 GB, one slot, lower power than 2× 3090 |
| Biggest models, quietly | Apple M4 Max 128GB or Apple M3 Ultra 512GB | Huge unified memory; slower per token |
| Big MoE models on a budget | Ryzen AI Max+ 395 128GB / NVIDIA DGX Spark 128GB | 96–112 GB usable; ideal for gpt-oss-120b |
Which models each memory size unlocks
- 8 GB: 7–8B models at Q4_K_M.
- 12 GB: 12–14B models; Flux in GGUF Q4–Q5.
- 16 GB: 14B at high quality, 24B at Q4, gpt-oss-20b natively.
- 24 GB: 27–32B models; the sweet spot for local AI.
- 48 GB: 70B models at Q4_K_M.
- 96 GB+: 70B at Q8, 100B+ MoE models.
Things to check before buying
- Get the right memory variant: the RTX 3060 and 4060 Ti exist in 8 GB versions.
- Check your power supply and case: a 3090 or 5090 needs a strong PSU and room for a large cooler.
- Two GPUs need two suitable PCIe slots; inference works fine at x4 or x8.
- For training, image tools and vLLM, NVIDIA's CUDA ecosystem is still the safest choice.
Compare specific cards: RTX 3090 vs 4090, RTX 4090 vs 5090, M4 Max vs RTX 5090.
New vs used
For local LLMs, used cards are often the best value. Inference is gentle on hardware compared with mining or rendering, and memory capacity does not wear out. The used RTX 3090 24GB is popular for that reason: it has the same 24 GB as a 4090, nearly the same bandwidth, and costs far less. When buying used, check that the card runs a long stress test without memory errors, that fans are quiet, and that you can see the seller's original receipt where possible.
Budget tiers at a glance
| Budget | Best LLM choice | What it runs | 8B speed |
|---|---|---|---|
| Entry | RTX 3060 12GB | 12–14B models | ~43 tok/s |
| Lower mid | RTX 5060 Ti 16GB | gpt-oss-20b, 24B at Q4 | ~53 tok/s |
| Mid | RTX 3090 24GB (used) | 27–35B models | ~109 tok/s |
| High | RTX 5090 32GB | 32B at Q6_K, long context | ~202 tok/s |
| Workstation | RTX PRO 6000 Blackwell 96GB | 70B at Q8_0, 120B MoE | ~202 tok/s |
| Quiet big-memory | Apple M4 Max 128GB | 120B-class MoE | ~64 tok/s |
Power, noise and the rest of the PC
- Power supply: plan for the GPU's full rated power plus 200–300 W for the rest of the system. Two 3090s need a quality 1,200 W unit or power limits.
- Power limits: LLM generation is limited by memory, not compute, so capping a 3090 or 4090 at 70–80% power costs only a few percent of speed while cutting heat and noise a lot.
- System RAM: 32 GB is a sensible minimum, 64 GB if you want to offload larger models.
- Storage: models are large. A 70B Q4 file is over 40 GB, so a 2 TB NVMe drive fills faster than you expect.
When a Mac or mini PC makes more sense
If you want to run 70B dense or 100B+ mixture-of-experts models quietly, without building a multi-GPU tower, unified-memory systems are hard to beat on capacity per watt. They are weaker at image generation, training and anything that needs fast prompt processing. See the Mac guide, M4 Max vs DGX Spark and DGX Spark vs Ryzen AI Max.
Cards to avoid for AI
8 GB cards at mid-range prices, such as the RTX 5060 Ti 8GB, are a poor buy for local AI; the 16 GB version of the same card costs a little more and runs far more. Very old datacenter cards like the Tesla P40 24GB are cheap but slow, loud without custom cooling and unsupported by many newer tools.