Why Does My Model Use More VRAM Than Expected?

Updated October 14, 2026 · Figures calculated live from our model data

The usual culprit is the context: many apps reserve the full context window at load time, and the KV cache can be larger than the model itself. Parallel request slots, a vision encoder, and vLLM's memory preallocation are the next most common causes.

1. The context is bigger than you think

Memory for the KV cache is reserved when the model loads, for the full context you set. Llama 3.1 8B needs 6.3 GB at 8K but 21.3 GB at its full 128K. Set the context explicitly (llama.cpp -c 8192, Ollama num_ctx, LM Studio's context slider) to what you actually need.

2. Parallel slots multiply the cache

Ollama (OLLAMA_NUM_PARALLEL) and llama.cpp server (-np) can serve several requests at once, and each needs its own cache. Four slots at 8K cost as much cache as one at 32K.

3. vLLM grabs most of the GPU on purpose

vLLM preallocates --gpu-memory-utilization (0.9 by default) of the card for weights and a KV cache pool. Seeing 90% usage is normal. Lower the value or set --max-model-len if you need room for other things.

4. Extra components

• Vision models load an image encoder (0.5–2 GB).
• Speculative decoding loads a second, draft model.
• Embedding or reranking models in the same app take memory too.

5. Your desktop is using the GPU

A browser, desktop compositor and video playback can take 0.5–1.5 GB on the GPU that drives your display. Plug the monitor into integrated graphics if you can, and close GPU-heavy apps.

6. Runtime differences

• Runtimes without sliding-window support store full caches for Gemma 3 27B and similar models.
• Without MLA support, DeepSeek R1 671B needs many times more cache memory.
• Flash attention reduces compute buffers; enable it where available.

Quick fixes for CUDA out of memory

1. Lower the context length.
2. Use a Q8_0 KV cache (needs flash attention in llama.cpp and Ollama).
3. Drop one quantization level.
4. Reduce parallel slots to 1.
5. Offload a few layers to the CPU (-ngl in llama.cpp).
6. For image models, enable tiled VAE decoding and model swapping.

Check the expected number for your exact setup with the LLM VRAM calculator or image model calculator.

Read your runtime's memory log

Most runtimes print exactly what they allocate when the model loads. In llama.cpp, look for lines like these:

load_tensors:        CUDA0 model buffer size = 15488.00 MiB
llama_kv_cache:      CUDA0 KV buffer size   =  2048.00 MiB
llama_context:       CUDA0 compute buffer size =   580.00 MiB

The three numbers map directly onto our calculator: weights, KV cache and overhead. If the KV buffer is much larger than you expected, the context is larger than you think. If the model buffer is larger, you have a bigger quantization than you think, or a vision projector is loaded too.

Checking actual usage

• NVIDIA: nvidia-smi shows memory per process; nvidia-smi -l 1 refreshes every second while you send a prompt.
• AMD: rocm-smi or amdgpu_top.
• Mac: Activity Monitor's GPU History, or sudo powermetrics --samplers gpu_power.
• Ollama: ollama ps shows the total and the CPU/GPU split.

Example: one model, four very different totals

The same Llama 3.1 8B at Q4_K_M, depending on settings:

Memory that grows during a session

If memory use rises over time rather than at load, the usual causes are image inputs in a vision model (each image is encoded into tokens), a runtime that grows its cache on demand rather than reserving it, or another program such as a browser tab using WebGPU. Restarting the runtime resets it; setting a fixed context stops on-demand growth.

Image and video generation

In ComfyUI and Forge, the classic out-of-memory error at the very end of a generation is the VAE decode step. Enable tiled VAE decoding. Errors at the start are usually the text encoder: load it in FP8 or let the app swap models. See running Flux on low VRAM.