How We Calculate VRAM

Updated October 14, 2026

Every number on this site comes from the same open formulas, applied to each model's published architecture (its config.json on Hugging Face). The JavaScript is not minified, and the data is public at /data/models.json and /data/gpus.json.

Units

Memory is shown in GB meaning GiB (1,073,741,824 bytes), the unit GPU vendors use for VRAM capacity (a "24 GB" card has 24 GiB). GGUF file sizes on the GGUF calculator are in decimal GB to match Hugging Face.

1. Weights

weights = parameters × bits per weight ÷ 8. Bits per weight are effective averages: Q8_0 8.5, Q6_K 6.59, Q5_K_M 5.69, Q4_K_M 4.85, IQ4_XS 4.25, Q3_K_M 3.91, Q2_K ≈3.0, FP8 8, FP16 16. GGUF k-quants keep some tensors at higher precision, so Q4_K_M is not 4.0 bits. Native releases (gpt-oss MXFP4, DeepSeek FP8, MiniMax FP8, Kimi INT4) use the official checkpoint size.

2. KV cache

KV = 2 × layers × KV heads × head dimension × bytes per value × tokens × sequences. Bytes per value: FP16 2, Q8_0 1.0625, Q4_0 0.5625.

  • Grouped-query attention is captured by using KV heads, not query heads.
  • Sliding-window layers (Gemma 3, gpt-oss, Llama 4 chunked attention) cache at most their window; only full-attention layers grow with context.
  • Multi-head Latent Attention (DeepSeek V3/R1, Kimi K2, Mistral Small 4) caches one compressed latent per layer per token (576 values for DeepSeek R1).
  • Hybrid linear attention (Qwen 3.5, 3.6, 3.8 and Qwen3-Next): only every fourth layer keeps a KV cache. The Gated DeltaNet layers hold a fixed FP32 state of layers × (value heads × dim² + 3 × (2 × key heads + value heads) × dim) values that does not grow with context (about 150 MB for Qwen 3.6 27B).
  • Shared-KV and mixed layers (Gemma 4): E2B and E4B reuse earlier layers' caches in their last 20 and 18 layers; global layers use fewer, larger KV heads than sliding layers. We count each layer group separately and assume keys and values are stored separately.
  • Compressed attention (DeepSeek V4): layers keep a 128-token window plus one compressed entry per 4 or per 128 tokens. These figures are estimates from config.json.

Example: Llama 3.1 8B has 32 layers, 8 KV heads and 128-dim heads: 2 × 32 × 8 × 128 × 2 bytes = 128 KB per token.

3. Runtime overhead

overhead = 0.5 GB + (vocab × 512 × 4 + hidden × 512 × 32) bytes, covering the CUDA/Metal context, logits and the compute buffer for a 512-token batch. This lands between 0.6 and 1.2 GB for most models, in line with llama.cpp logs.

4. Will it fit?

Usable memory is the card's VRAM, or for unified-memory systems the default GPU allocation (about two thirds of RAM on 16–36 GB Macs, 75% on larger Macs; 96 GB on Ryzen AI Max+ 395; 112 GB on DGX Spark). Status:

  • Fits: total ≤ 94% of usable memory (leaves room for the display and fragmentation).
  • Tight fit: between 94% and 100%.
  • Partial offload (discrete GPUs): at least 30% of the model fits; the rest runs from system RAM.
  • Too small: less than that, or any overflow on unified memory.

5. Tokens per second

Single-user generation is memory-bound: seconds per token = (active weights + half the KV cache) ÷ (bandwidth × efficiency) + layers × overhead, with tokens/s the inverse. The overhead is 0.01 ms per layer for dense models and 0.1 ms per layer for mixture-of-experts models (expert routing and many small kernels), calibrated against published llama.cpp results. Efficiency is 0.65 for NVIDIA and Apple, 0.55 for AMD discrete GPUs, 0.6 for APUs and 0.5 for Intel. For partial offload, the CPU part uses 60 GB/s × 0.7 (dual-channel DDR5). Treat speeds as ±25%; prompt processing is not estimated.

6. Fine-tuning

  • Base weights: 2 bytes/parameter (BF16), or 0.516 for QLoRA (NF4 with double quantization).
  • Full fine-tuning states: 14 bytes/parameter with AdamW (BF16 gradients, FP32 master weights, two FP32 moments) or 8 with 8-bit Adam.
  • LoRA: rank × (in + out) per targeted matrix, at 16 bytes per trainable parameter (FP32 weights, gradients and Adam moments).
  • Activations per layer: seq × batch × (10 × hidden + 8 × intermediate) bytes without checkpointing; with checkpointing, 2 × seq × batch × hidden per layer plus one layer recomputed. Logits: seq × batch × vocab × 6 bytes.
  • Overhead: 1 GB + 8% allocator fragmentation.

7. Image and video models

Weights for the diffusion model, text encoders and VAE are calculated from parameter counts. Latent tokens = (width ÷ patch) × (height ÷ patch) × latent frames. Activation memory is a calibrated estimate: tokens × hidden × 2 bytes × CFG passes × 8. VAE decode uses about 3 GB per megapixel (×4 frames for video), divided by 4 with tiling. "With swapping" takes the largest single stage; "all loaded" sums the weights. Treat these as ±20%.

Known limitations

  • Runtimes differ: vLLM preallocates memory, and older runtimes may not support sliding windows or MLA.
  • Vision encoders, draft models and LoRA adapters loaded at inference are not included.
  • Speeds assume a single user and a fully loaded GPU.

Found a figure that differs from what you measure? Tell us with your runtime, settings and log output, and we will check it and record any fix in the changelog.