Best for Local inference
Best open-source AI models for local inference
Running models locally means trading frontier quality for control: no API limits, no data leakage, no monthly bills. The sweet spot for most users in 2026 is 8B–32B parameters at Q4 quantization, which fits on a single 24 GB GPU or a Mac with 32 GB of unified memory.
We're optimising for quality at Q4 VRAM under 20 GB, fast tokens-per-second on consumer GPUs, and ecosystem support across Ollama, llama.cpp, LM Studio, and MLX.
Local inference is the original promise of open-source AI: full control of your stack. A model that runs slowly defeats the purpose.
Our picks
The default local-inference workhorse. ~6 GB at Q4, runs on any 12 GB+ GPU.
Apache 2.0 and slightly stronger than Llama 3.1 8B on coding.
14B with frontier-class reasoning. ~9 GB at Q4.
Highest-quality model that still fits on a 4090 at Q4 (~20 GB).
Strong default if you can't accept Llama's community-licence terms.
Pocket-sized fallback for laptops and on-device deployment.
Things to watch out for
- KV cache memory grows linearly with context. For 128K-context workloads, budget 1.5–2× the listed VRAM.
- Apple Silicon users: MLX and llama.cpp Metal are the fastest paths. M3/M4 Max with 64 GB unified memory can comfortably run 32B at Q4.
- For CPU-only inference, prefer models under 8B parameters. Q4 quantization is essentially mandatory.
All picks at a glance
The workhorse 8B instruction-tuned model. Excellent quality-to-cost ratio and the broadest ecosystem support of any open-weights model — every major inference engine, fine-tuning library, and quantization toolchain has a 3.1 8B preset. Fits in 24 GB of VRAM at fp16, ~6 GB at Q4. Strong default for production chat where 70B is overkill, for fine-tuning on a specialist task, and for any workload where you want a known-good baseline.
- Context
- 128K
- License
- llama-3
- VRAM Q4
- 4.8 GB
Apache-2.0-licensed 7B model with surprisingly strong reasoning and multilingual chops. Qwen 2.5 trains on a larger and more carefully filtered corpus than the original Qwen series, and the 7B variant punches well above its weight on coding and math benchmarks. A strong default for cost-sensitive chat workloads and for fine-tuning experiments where the Apache licence simplifies downstream redistribution.
- Context
- 128K
- License
- apache-2-0
- VRAM Q4
- 4.2 GB
14B model trained primarily on synthetic data. Punches above its weight on reasoning, especially MATH and GPQA. MIT licensed. A standout choice when you want strong reasoning quality without paying 70B-tier hardware costs. Phi-4 in particular demonstrated that careful synthetic-data curation can extract frontier-class reasoning from a relatively small dense model.
- Context
- 16K
- License
- mit
- VRAM Q4
- 8.4 GB
32B sweet-spot model: strong reasoning, fits on one H100 in fp16, on a 4090 at Q4. The 32B size in particular hits a quality/cost knee — quality scales with parameters faster than cost up to ~32B, and slower afterwards. Favoured for production chat where 7B isn't sharp enough and where 70B+ would over-spec the hardware budget. Apache 2.0 licence.
- Context
- 128K
- License
- apache-2-0
- VRAM Q4
- 19.2 GB
Mid-tier Gemma. Strong general-purpose chat model at small scale. The Gemma Terms of Use permit commercial use subject to Google's prohibited-use policy.
- Context
- 8K
- License
- gemma
- VRAM Q4
- 5.4 GB
Pocket-sized Llama 3 variant for edge deployment. Surprising chat quality after instruction tuning makes it competitive with much larger models from a previous generation. At Q4 it fits in ~2 GB of VRAM and runs on consumer GPUs and recent Apple Silicon. A strong default for on-device chat, summarisation, and structured extraction tasks where the workload doesn't need frontier reasoning quality.
- Context
- 128K
- License
- llama-3
- VRAM Q4
- 1.8 GB