Best Local Coding Models for a 24 GB GPU
For coding agents on a 24 GB GPU, context length matters as much as model quality. Agents such as Cline, Roo Code, Continue and Aider send 20K–100K tokens of files and history. Qwen 3.6 27B and Qwen 3.6 35B-A3B hold far longer contexts than older 32B models thanks to hybrid attention.
Models compared on 24 GB
Q4_K_M (gpt-oss-20b in native MXFP4). "Max context" is the longest that fits 24 GB with a 6% margin; speed is estimated on an RTX 3090 24GB at 32K context.
| Model | VRAM at 8K | Max context, FP16 cache | Max context, Q8_0 cache | Speed |
|---|---|---|---|---|
| Qwen 3.6 27B | 16.9 GB | 98K | 185K | ~34 tok/s |
| Qwen 3.6 35B-A3B | 21.0 GB | 88K | 166K | ~131 tok/s |
| Qwen3-Coder 30B-A3B | 18.8 GB | 48K | 90K | ~93 tok/s |
| Gemma 4 31B | 19.8 GB | 43K | 89K | ~29 tok/s |
| gpt-oss-20b | 13.9 GB | 128K | 128K | ~144 tok/s |
| Mistral Small 3.1 24B | 15.6 GB | 52K | 98K | ~35 tok/s |
| Qwen 2.5 Coder 32B | 21.4 GB | 12K | 23K | ~7.4 tok/s |
Our picks
- Best overall: Qwen 3.6 27B. A dense model with strong coding results and room for about 98K tokens of context with an FP16 cache.
- Fastest: Qwen 3.6 35B-A3B or Qwen3-Coder 30B-A3B. Only about 3B parameters are active per token, so they respond quickly even with long prompts.
- Alternative: Gemma 4 31B, with a lean cache from sliding-window layers.
- Most headroom: gpt-oss-20b, small enough to keep its full 128K context with memory to spare.
Why older 32B models struggle with agents
Qwen 2.5 Coder 32B was the local coding favourite, but every one of its 64 layers stores keys and values: 32K of context costs 8.0 GB in FP16. On 24 GB that leaves room for only about 12K tokens. Newer hybrid models need 2.1 GB for the same 32K.
Settings for coding agents
- Set the context explicitly to what your agent uses (32K–64K is typical). See Ollama VRAM settings.
- Use a Q8_0 KV cache to roughly double the usable context.
- Keep one parallel slot; agents send one request at a time.
- Prefer Q4_K_M or Q5_K_M over lower quants; small errors compound over multi-step edits.
With 32 GB (RTX 5090 32GB) you can step up to Q6_K or Q8_0 for the same models. With 16 GB, gpt-oss-20b is the strongest choice that fits fully.
Setting up a coding agent with a local model
- Run the model as a server:
ollama serve, LM Studio's server, orllama-server. - Set the context explicitly. For Qwen 3.6 27B on 24 GB, 64K with an FP16 cache uses 20.4 GB.
- In your editor tool (Continue, Cline, Roo Code, Aider), choose the OpenAI-compatible or Ollama provider and point it at the local server.
- Match the tool's context setting to the server's, so it does not send more than the model can hold.
llama-server -m Qwen3.6-27B-Q4_K_M.gguf -ngl 99 -c 65536 -fa on \
--jinja --port 8080The --jinja flag uses the model's own chat template, which many tool-calling agents need.
Autocomplete is a different job
Tab completion needs very low latency rather than deep reasoning. A small model such as Qwen 3.5 4B or Llama 3.2 3B works better than a 27B model for this. You can run both: a small completion model and a larger chat or agent model, as long as their combined memory fits.
What to expect
Local 27–35B models handle everyday coding tasks well: explaining code, writing functions and tests, refactoring within a file, and fixing clear bugs. They are weaker than the best hosted models at large multi-file changes and long autonomous runs. Using an agent's plan mode, keeping tasks small and giving clear file context makes a big difference.
Speed matters more than you think
Agents make many calls per task. At ~34 tokens/s a long agent run can take minutes; at ~131 tokens/s it feels interactive. Many people use the fast mixture-of-experts model for agent loops and switch to the dense model for harder single questions.