Is Q4_K_M good enough?

For most chat and coding use, yes. It is the default for most GGUF downloads because the quality loss is small relative to the memory saved.

Use Q5_K_M if it fits with the context you need; otherwise Q4_K_M. The difference is small, while running out of memory is a big problem.

Does quantization make models faster?

For generation, yes: fewer bytes per weight means less data to read per token, and generation speed is limited by memory bandwidth.

Which quantization should I download?

Q4_K_M is the safe default. Use Q5_K_M or Q6_K if they fit with your context, and IQ4_XS or Q3_K_M only to fit a larger model than you could otherwise run.

Does the KV cache use the same quantization as the model?

No. The cache has its own setting, FP16 by default. You can set it to Q8_0 separately in llama.cpp, Ollama and LM Studio.

Q4 vs Q5 vs Q8: Quantization Explained

Updated October 14, 2026 · Figures calculated live from our model data

Quantization stores model weights with fewer bits. Q8_0 is nearly lossless, Q4_K_M is the usual sweet spot, and below Q4 quality drops quickly. On Llama 3.1 8B, going from FP16 to Q4_K_M cuts the weights from 15.0 GB to 4.5 GB.

What the names mean

In GGUF (the format used by llama.cpp, Ollama and LM Studio), the number is the base bits per weight. K means k-quants, which group weights in blocks with their own scales. The letter at the end is the mix: S, M and L keep progressively more sensitive tensors (attention, output) at higher precision. That is why Q4_K_M averages about 4.85 bits, not 4.

Memory and quality at each level

Sizes for Qwen 3 32B with 8K context. Quality notes reflect common perplexity and benchmark comparisons; exact loss varies by model.

Format	Bits/weight	Total VRAM	Quality
FP16 / BF16	16	64.0 GB	Reference
Q8_0	8.5	35.3 GB	Practically identical to FP16
Q6_K	6.59	28.0 GB	Very close to Q8_0
Q5_K_M	5.69	24.6 GB	Small loss, hard to notice in chat
Q4_K_M	4.85	21.4 GB	Usual sweet spot; small measurable loss
IQ4_XS	4.25	19.1 GB	Close to Q4_K_M, smaller
Q3_K_M	3.91	17.8 GB	Noticeable loss, more errors
Q2_K	≈3.0	14.3 GB	Large loss; last resort

Bigger model at low bits, or smaller model at high bits?

Larger models tolerate quantization better. A 32B model at Q4_K_M almost always beats a 14B model at Q8_0, and the two need similar memory. The rule breaks down around Q3 and below, where a smaller model at Q4–Q5 is often the better choice.

Other formats

GPTQ and AWQ: 4-bit GPU formats for vLLM and Transformers. Size is similar to Q4_K_M; speed is higher for batched serving.
EXL2 / EXL3: ExLlama formats with any bitrate (for example 4.65 bpw), good for fitting a model exactly into 24 GB.
FP8: 8-bit floating point, native on Ada, Hopper and Blackwell GPUs. Common for serving in vLLM.
MXFP4 / NVFP4: 4-bit floating-point formats. gpt-oss-20b ships natively in MXFP4.

Head-to-head comparisons

Q4_K_M vs Q5_K_M, Q4_K_M vs Q8_0, IQ4_XS vs Q4_K_M, FP8 vs Q8_0 and AWQ vs GGUF show the memory difference on ten popular models.

Don't forget the KV cache

You can quantize the context cache too. Q8_0 halves it with very little loss; Q4_0 saves more but can hurt long-context recall. See the KV cache calculator.

How to choose a quantization in practice

Start from the memory you have, minus about 6% for the desktop and buffers.
Subtract the KV cache for the context you need. For long contexts this matters as much as the model.
Pick the highest quantization whose weights fit in what is left. Q6_K or Q5_K_M if they fit, Q4_K_M otherwise.
If even Q4_K_M does not fit, compare two options: IQ4_XS or Q3_K_M of the same model, or Q4_K_M of the next size down. For models above about 30B the first is usually better; below that, the second.

Example: one 24 GB card, four choices

With an RTX 3090 24GB and a 16K context:

Option	VRAM	Fits?
Qwen 3 32B at Q4_K_M	23.4 GB	Tight fit
Qwen 3 32B at Q5_K_M	26.6 GB	85% on GPU
Qwen 3 14B at Q8_0	18.0 GB	Fits
Qwen 3.6 27B at Q6_K	22.8 GB	Tight fit

The larger model at a lower bit width is usually the stronger choice, as long as it fits without offloading.

Why small models suffer more

Quantization adds a little noise to every weight. A large model has more redundancy, so the same noise costs it less. On models under about 8B, Q4 is noticeably weaker than Q8 for maths, code and non-English text, so Q5_K_M or Q6_K is worth it when memory allows. On 70B models, Q4_K_M is close to the original for most tasks.

Importance matrices and "imatrix" files

Many GGUF uploads now use an importance matrix: a measurement, taken on sample text, of which weights matter most. The quantizer spends more precision on those weights. Imatrix files of the same type and size are usually a little better than plain ones. Lower-bit i-quants (IQ3, IQ2) depend on it and should only be used from uploaders who generate one.

Where to compare formats on real models

Our quantization comparison pages put the numbers side by side for ten popular models: Q4_K_M vs Q5_K_M, Q5_K_M vs Q6_K, Q3_K_M vs Q4_K_M and FP16 vs Q8_0. The GGUF size calculator shows the file size of every format for any model.