What LLMs Can the RTX 4060 8GB Run?

The RTX 4060 8GB has 8 GB of VRAM and 272 GB/s of bandwidth. It runs 18 of 58 models fully at Q4_K_M with 8K context, up to Qwen 3.5 9B, and generates about 32 tokens/s on an 8B model.

NVIDIA · Ada Lovelace · Released 2023-06 · Updated October 14, 2026

Your GPU or Mac

Quantization

Context length

KV cache precision

18 of 58 models fit

RTX 4060 8GB (8.0 GB) at Q4_K_M with 8K context. Largest that fits fully: Qwen 3.5 9B.

Model	Needs	Status	Speed	Best quant that fits
Ministral 3 8B	6.7 GB	Fits	~31 tok/s	Q5_K_M
Qwen 3 8B	6.6 GB	Fits	~31 tok/s	Q5_K_M
Qwen 3.5 9B	6.4 GB	Fits	~31 tok/s	Q5_K_M
Llama 3.1 8B	6.3 GB	Fits	~32 tok/s	Q5_K_M
DeepSeek R1 Distill Llama 8B	6.3 GB	Fits	~32 tok/s	Q5_K_M
Mistral 7B v0.3	5.7 GB	Fits	~35 tok/s	Q6_K
Gemma 4 E4B	5.7 GB	Fits	~35 tok/s	Q6_K
Qwen 2.5 7B	5.6 GB	Fits	~36 tok/s	Q6_K
DeepSeek R1 Distill Qwen 7B	5.6 GB	Fits	~36 tok/s	Q6_K
Qwen 3 4B	4.2 GB	Fits	~57 tok/s	Q8_0
Phi-4 Mini	4.1 GB	Fits	~61 tok/s	Q8_0
Gemma 4 E2B	4.0 GB	Fits	~56 tok/s	Q8_0
Gemma 3 4B	3.7 GB	Fits	~62 tok/s	Q8_0
Qwen 3.5 4B	3.7 GB	Fits	~63 tok/s	Q8_0
Ministral 3 3B	3.5 GB	Fits	~69 tok/s	Q8_0
Llama 3.2 3B	3.5 GB	Fits	~72 tok/s	Q8_0
Llama 3.2 1B	1.7 GB	Fits	~193 tok/s	FP16 / BF16
Gemma 3 1B	1.6 GB	Fits	~260 tok/s	FP16 / BF16
Qwen 2.5 32B	21.4 GB	35% on GPU	~2.7 tok/s	–
Qwen 2.5 Coder 32B	21.4 GB	35% on GPU	~2.7 tok/s	–
DeepSeek R1 Distill Qwen 32B	21.4 GB	35% on GPU	~2.7 tok/s	–
Qwen 3 32B	21.4 GB	35% on GPU	~2.7 tok/s	–
Qwen 3.5 35B-A3B	21.0 GB	36% on GPU	~26 tok/s	–
Qwen 3.6 35B-A3B	21.0 GB	36% on GPU	~26 tok/s	–
Gemma 4 31B	19.8 GB	38% on GPU	~3.0 tok/s	–
Qwen 3 30B-A3B	18.8 GB	40% on GPU	~22 tok/s	–
Qwen3-Coder 30B-A3B	18.8 GB	40% on GPU	~22 tok/s	–
Gemma 3 27B	17.6 GB	43% on GPU	~3.6 tok/s	–
Qwen 3.5 27B	16.9 GB	45% on GPU	~3.8 tok/s	–
Qwen 3.6 27B	16.9 GB	45% on GPU	~3.8 tok/s	–
Qwen 3.8 27B	16.9 GB	45% on GPU	~3.8 tok/s	–
Mistral Small 3.1 24B	15.6 GB	48% on GPU	~4.4 tok/s	–
Gemma 4 26B-A4B	15.6 GB	48% on GPU	~24 tok/s	–
gpt-oss-20b	12.9 GB	58% on GPU	~31 tok/s	–
Qwen 2.5 14B	10.7 GB	70% on GPU	~9.3 tok/s	Q2_K
DeepSeek R1 Distill Qwen 14B	10.7 GB	70% on GPU	~9.3 tok/s	Q2_K
Phi-4 14B	10.6 GB	71% on GPU	~9.3 tok/s	Q2_K
Qwen 3 14B	10.5 GB	72% on GPU	~9.6 tok/s	Q2_K
Ministral 3 14B	9.7 GB	78% on GPU	~12 tok/s	Q2_K
Mistral Nemo 12B	9.0 GB	84% on GPU	~14 tok/s	Q2_K
Gemma 3 12B	8.8 GB	86% on GPU	~15 tok/s	Q3_K_M
Kimi K2.6	566 GB	Too small	–	–
DeepSeek R1 671B	380 GB	Too small	–	–
Llama 3.1 405B	234 GB	Too small	–	–
Qwen 3.5 397B-A17B	226 GB	Too small	–	–
DeepSeek V4 Flash	161 GB	Too small	–	–
Qwen 3 235B-A22B	135 GB	Too small	–	–
MiniMax-M2	133 GB	Too small	–	–
Qwen 3.5 122B-A10B	70.2 GB	Too small	–	–
Mistral Small 4 119B	68.2 GB	Too small	–	–
gpt-oss-120b	67.2 GB	Too small	–	–
Llama 4 Scout	64.0 GB	Too small	–	–
GLM-4.5-Air	62.1 GB	Too small	–	–
Qwen3-Next 80B-A3B	46.3 GB	Too small	–	–
Qwen 2.5 72B	44.5 GB	Too small	–	–
Llama 3.3 70B	43.2 GB	Too small	–	–
DeepSeek R1 Distill Llama 70B	43.2 GB	Too small	–	–
Mixtral 8x7B	28.0 GB	Too small	–	–

RTX 4060 8GB for local AI

An efficient 8 GB card. A narrow 128-bit bus limits token speed, and 8 GB limits you to 8B-class models at Q4_K_M with moderate context.

Among the 9 devices we track with about 8 GB of usable memory, the RTX 4060 8GB ranks 8 of 9 for text-generation speed, because generation speed follows memory bandwidth. Others in the same class: RTX 3070 Ti 8GB, RTX 3060 Ti 8GB, RTX 3070 8GB, RTX 5060 8GB. They all hold the same models; the faster ones simply answer sooner.

Our picks for the RTX 4060 8GB

The model we would run first for each job, chosen from our recommendations by what fits fully with an 8K context. "Best quant" is the highest quality that fits; "max context" is at Q4_K_M or the native format.

Task	Model	Best quant	Max context	Speed
Best all-rounder	Qwen 3.5 9B	Q5_K_M	44K	~31 tok/s
Coding	Qwen 3.5 9B	Q5_K_M	44K	~31 tok/s
Fastest replies	Qwen 3.5 4B	Q8_0	130K	~63 tok/s
Long documents (64K+)	Gemma 4 E4B	Q6_K	124K	~35 tok/s
Images in, text out	Gemma 4 E4B	Q6_K	124K	~35 tok/s
Image generation	FLUX.1 [dev]	GGUF Q4_K_S / NF4	–	–

Spec	Value
Memory	8 GB
Memory bandwidth	272 GB/s
Architecture	Ada Lovelace
Released	2023-06
Llama 3.1 8B Q4_K_M speed	~32 tok/s
Largest model that fits	Qwen 3.5 9B

Image and video models on the RTX 4060 8GB

Peak VRAM with model swapping (ComfyUI default on smaller cards).

Model	FP8	FP8 fit	GGUF Q4	Q4 fit
Stable Diffusion 1.5	1.8 GB	Fits	1.7 GB	Fits
SDXL 1.0	4.1 GB	Fits	4.1 GB	Fits
Stable Diffusion 3.5 Medium	6.4 GB	Fits	6.4 GB	Fits
Stable Diffusion 3.5 Large	8.6 GB	87% on GPU	6.4 GB	Fits
FLUX.1 [dev]	12.1 GB	62% on GPU	7.3 GB	Fits
FLUX.1 [schnell]	12.1 GB	62% on GPU	7.3 GB	Fits
Qwen-Image	20.4 GB	37% on GPU	12.3 GB	61% on GPU
Wan 2.2 A14B (T2V / I2V)	18.8 GB	40% on GPU	13.2 GB	57% on GPU
Wan 2.2 TI2V 5B	8.0 GB	Tight fit	6.5 GB	Fits
HunyuanVideo	18.3 GB	41% on GPU	13.1 GB	57% on GPU
FLUX.1 Kontext [dev]	12.3 GB	61% on GPU	7.5 GB	Fits
FLUX.2 [dev]	31.0 GB	Too small	23.6 GB	32% on GPU
Z-Image Turbo	6.8 GB	Fits	4.9 GB	Fits
Qwen-Image-Edit 2509	20.5 GB	37% on GPU	12.4 GB	61% on GPU
LTX-2	19.2 GB	39% on GPU	12.6 GB	60% on GPU

Upgrade path

The next step up is the Intel Arc B570 10GB with 10 GB and 40% more bandwidth.

RTX 4060 8GB for LLMs: common questions

Can the RTX 4060 8GB run Llama 3.3 70B?

No. It needs about 43.2 GB at Q4_K_M, far more than the 8.0 GB available.

Can the RTX 4060 8GB run Qwen 3 32B?

Only partly. It needs about 21.4 GB at Q4_K_M, so about 35% fits in VRAM and the rest runs from system RAM, which is slow.

Can the RTX 4060 8GB run Qwen 3 14B?

Only partly. It needs about 10.5 GB at Q4_K_M, so about 72% fits in VRAM and the rest runs from system RAM, which is slow. Q2_K fits fully.

How fast is the RTX 4060 8GB for LLMs?

About 32 tokens per second for an 8B model at Q4_K_M, based on its 272 GB/s memory bandwidth. Larger models scale down roughly in proportion to their size.