Tokens per Second Estimator for Local LLMs
Estimate text generation speed from memory bandwidth, model size and quantization. Llama 3.1 8B at Q4_K_M runs at roughly 117 tokens/s on an RTX 4090 and 43 tokens/s on an RTX 3060 12GB.
~117 tokens/sec
Llama 3.1 8B at Q4_K_M on the RTX 4090 24GB Fits
- Data read per token5.41 GB
- Weights read4.87 GB
- KV cache read (average)0.54 GB
- Bandwidth × efficiency1008 × 0.65 = 655 GB/s
- Per-token layer overhead0.3 ms
Same model on other hardware
- RTX 3060 12GBFits~43 tok/s
- RTX 4060 Ti 16GBFits~34 tok/s
- RTX 3090 24GBFits~109 tok/s
- Radeon RX 7900 XTX 24GBFits~95 tok/s
- RTX 4090 24GBFits~117 tok/s
- RTX 5090 32GBFits~202 tok/s
- RTX 6000 Ada 48GBFits~111 tok/s
- A100 80GBFits~227 tok/s
- H100 80GB SXMFits~357 tok/s
- Ryzen AI Max+ 395 128GBFits~28 tok/s
- Apple M4 Max 128GBFits~64 tok/s
- Apple M5 Max 128GBFits~72 tok/s
- RTX PRO 6000 Blackwell 96GBFits~202 tok/s
- NVIDIA DGX Spark 128GBFits~30 tok/s
- Apple M3 Ultra 512GBFits~95 tok/s
Why bandwidth sets the speed
To generate each token, the GPU reads every active weight (plus the KV cache) from memory once. Generation is therefore limited by memory bandwidth, not compute:
time per token ≈ GB read per token ÷ (bandwidth × efficiency) + layer overhead
Efficiency is about 0.65 for NVIDIA and Apple in llama.cpp, 0.55 for AMD and 0.5 for Intel. This is why a used RTX 3090 (936 GB/s) generates nearly as fast as an RTX 4090 (1,008 GB/s), and why mixture-of-experts models are fast: they only read their active parameters. The layer overhead (about 0.01 ms per layer, or 0.1 ms for MoE layers because of expert routing) keeps small-active MoE estimates realistic.
What this does not cover
Prompt processing (reading your input) is compute-bound and much faster per token. Batched serving in vLLM multiplies total throughput. Speculative decoding can add 1.5–2.5× on top. Estimates are ±25%.
Frequently asked questions
What is a good tokens per second for local LLMs?
Around 10 tokens/s feels like reading speed; 30+ feels instant for chat. Reasoning models generate long traces, so they benefit from 40+ tokens/s.
Why is my speed much lower than estimated?
The usual cause is part of the model spilling into system RAM, often because the context is larger than expected. Check your runtime log for the number of layers offloaded to the GPU.