Running Local LLMs on AMD GPUs
AMD GPUs run local LLMs well through llama.cpp-based apps (Ollama, LM Studio, llama.cpp) using ROCm or Vulkan. Memory needs are identical to NVIDIA; speed is somewhat lower per GB/s of bandwidth, and training and image tools remain more work to set up. AMD's advantage is more VRAM per dollar.
AMD cards for local AI
Largest model released since 2025 that fits fully at Q4_K_M with an 8K context, and estimated speed on Llama 3.1 8B.
| GPU | VRAM | Bandwidth | Runs | 8B speed |
|---|---|---|---|---|
| Radeon RX 7600 8GB | 8.0 GB | 288 GB/s | Qwen 3.5 9B | ~29 tok/s |
| Radeon RX 6700 XT 12GB | 12.0 GB | 384 GB/s | Qwen 3 14B | ~39 tok/s |
| Radeon RX 9060 XT 16GB | 16.0 GB | 320 GB/s | Gemma 4 26B-A4B | ~32 tok/s |
| Radeon RX 7800 XT 16GB | 16.0 GB | 624 GB/s | Gemma 4 26B-A4B | ~62 tok/s |
| Radeon RX 9070 XT 16GB | 16.0 GB | 640 GB/s | Gemma 4 26B-A4B | ~64 tok/s |
| Radeon RX 7900 XT 20GB | 20.0 GB | 800 GB/s | Gemma 4 31B | ~79 tok/s |
| Radeon RX 7900 XTX 24GB | 24.0 GB | 960 GB/s | Qwen 3.5 35B-A3B | ~95 tok/s |
| Radeon AI PRO R9700 32GB | 32.0 GB | 640 GB/s | Qwen 3.5 35B-A3B | ~64 tok/s |
| Ryzen AI Max+ 395 128GB | 96.0 GB | 256 GB/s | Qwen 3.5 122B-A10B | ~28 tok/s |
ROCm or Vulkan?
- ROCm is AMD's CUDA equivalent. It is officially supported on recent cards (RX 7900 series, RX 9000 series, Radeon AI PRO, Instinct) on Linux, with growing Windows support. Older RDNA 2 cards often work with an override such as
HSA_OVERRIDE_GFX_VERSION=10.3.0. - Vulkan works on almost any AMD (or Intel, or NVIDIA) GPU with no extra install, and llama.cpp's Vulkan backend is now competitive with ROCm for token generation. LM Studio ships both.
What works
| Tool | AMD support |
|---|---|
| Ollama | Yes (ROCm) on supported cards, Linux and Windows |
| LM Studio | Yes (ROCm and Vulkan runtimes) |
| llama.cpp | Yes (HIP/ROCm, Vulkan) |
| vLLM | Yes on ROCm, best on Instinct and RX 7900 / 9000 series |
| ComfyUI (images, video) | Yes with ROCm PyTorch on Linux; slower than NVIDIA |
| Training (Unsloth, Axolotl) | Partial; more setup than CUDA |
AMD vs NVIDIA at the same VRAM
Compare: RTX 4090 vs RX 7900 XTX, RTX 3090 vs RX 7900 XTX, RTX 5070 Ti vs RX 9070 XT, RTX 5060 Ti vs RX 9060 XT.
Strix Halo: AMD's unified-memory option
The Ryzen AI Max+ 395 128GB gives the GPU up to about 96 GB of shared memory, enough for gpt-oss-120b at ~43 tokens/s. It is one of the cheapest ways to run 100B-class MoE models.
Getting started on Linux with ROCm
- Install a distribution and kernel supported by the current ROCm release (Ubuntu LTS is the most common).
- Install ROCm with AMD's installer, add your user to the
renderandvideogroups, and reboot. - Check the GPU is visible with
rocminfoorrocm-smi. - Install Ollama, or build llama.cpp with HIP support.
# llama.cpp with ROCm (HIP)
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100
cmake --build build -j
# or the Vulkan backend, which needs no ROCm
cmake -B build -DGGML_VULKAN=ONgfx1100 is the RX 7900 series; use your card's target.
Windows
LM Studio is the easiest route on Windows, with ROCm and Vulkan runtimes built in. Ollama supports many Radeon cards on Windows directly. For ComfyUI and PyTorch-based tools, Linux remains more reliable for AMD.
Why AMD is slower per GB/s
Token generation reads weights from memory on both brands, but llama.cpp's CUDA kernels have had more optimisation work, and NVIDIA's memory subsystem tends to reach a higher share of its rated bandwidth. We model AMD discrete GPUs at slightly lower efficiency. The gap has narrowed with recent ROCm and Vulkan releases, and on RDNA 4 cards it is small.
Value comparison
The Radeon RX 7900 XTX 24GB offers 24 GB new for less than NVIDIA's 24 GB options, and the Radeon AI PRO R9700 32GB brings 32 GB to a workstation card at a lower price than NVIDIA's 32 GB cards. If your work is mainly chat, coding assistants and document analysis in llama.cpp-based apps, that extra memory per dollar often matters more than the speed difference.