Which quantization should I pick?

Quantization stores model weights at fewer bits to save VRAM. The trade-off is a small quality drop that's usually invisible at Q4_K_M and up. Here's the decision tree, plus a lookup table for every model in the catalogue on a 24 GB RTX 4090.

Decision tree

Do you have >= 2 × params in GB of VRAM? Run fp16. No quantization trade-off.
Do you have ≈ 1 × params in GB? Q8_0 — near-lossless with half the memory of fp16.
Do you have ≈ 0.7 × params in GB? Q5_K_M — quality drop under 1 % on most benchmarks.
Do you have ≈ 0.6 × params in GB? Q4_K_M. This is the mainstream default and what Ollama defaults to.
Anything less? Try a smaller model at Q4 first — it'll almost always beat a heavily-quantized larger one at real workloads.

Quantization levels compared

Level	Bits / weight	GB per B params	Typical quality loss	When to use
fp16	16	2.00	None (baseline).	Production inference on H100/H200 where memory isn't the bottleneck. Fine-tuning.
fp8	8	1.00	Negligible on H100+ hardware.	Production hosted inference. Doubles throughput vs fp16 with no measurable quality drop on Hopper/Blackwell.
Q8_0	8	1.00	≈ 0.1 % on academic benchmarks.	Highest-quality local option. Use when you have the VRAM headroom (2× the params in GB).
Q6_K	6.6	0.83	≈ 0.3 % typical.	Sweet spot between Q8 and Q5. Rarely the default, but useful when Q5 feels too aggressive.
Q5_K_M	5.5	0.69	≈ 1 % typical.	One notch above the mainstream default. Choose when you have 30–40 % more VRAM than Q4 needs.
Q4_K_M	4.85	0.60	≈ 1–3 % typical.	The mainstream local-inference default. 4× smaller than fp16 with modest quality cost.
Q3_K_M	3.9	0.49	≈ 3–6 %; visible on hard prompts.	Emergency only — when the model won't fit at Q4. Consider dropping to a smaller model instead.
Q2_K	2.6	0.33	≈ 6–12 %; clearly degraded.	Testing only. Don't run this in production.

Recommended quantization on a 24 GB RTX 4090

Assumes 25 % of VRAM reserved for KV cache + runtime overhead. Real-world numbers vary with context length; leave slack for long contexts.

Model	Params	Recommended
Llama 3.2 1B	1B	fp16
Gemma 2 2B	2.6B	fp16
Llama 3.2 3B	3B	fp16
Phi-3 Mini 4K Instruct	3.8B	fp16
OLMo 2 7B	7B	fp16
Llama 2 7B Chat	7B	fp16
Falcon 3 7B Instruct	7B	fp16
Falcon Mamba 7B	7B	fp16
Mistral 7B v0.3	7B	fp16
Qwen2.5 7B Instruct	7B	fp16
Hermes 3 Llama 3.1 8B	8B	fp16
Qwen 3 8B	8B	fp16
Llama 3.1 8B Instruct	8B	fp16
Gemma 2 9B	9B	fp16
Llama 3.2 11B Vision	11B	fp8
Mistral Nemo 12B	12B	fp8
Stable LM 2 12B	12B	fp8
OLMo 2 13B	13B	fp8
Llama 2 13B Chat	13B	fp8
Phi-4 14B	14B	fp8
Phi-3 Medium 14B	14B	fp8
Qwen2.5 14B Instruct	14B	fp8
Llama 4 Scout 17B (16E)	17B	fp8
Llama 4 Maverick 17B (128E)	17B	fp8
Mistral Small 3	24B	Q5_K_M
Gemma 2 27B	27B	Q4_K_M
QwQ 32B Preview	32B	Q3_K_M
Qwen2.5 Coder 32B	32B	Q3_K_M
Qwen 3 32B	32B	Q3_K_M
Qwen2.5 32B Instruct	32B	Q3_K_M
Yi VL 34B	34B	Q3_K_M
Yi 1.5 34B Chat	34B	Q3_K_M
Command R	35B	Q3_K_M
Mixtral 8×7B Instruct	46.7B	Q2_K
Llama 2 70B Chat	70B	won't fit
Llama 3.1 70B Instruct	70B	won't fit
Hermes 3 Llama 3.1 70B	70B	won't fit
Llama 3.1 Nemotron 70B Instruct	70B	won't fit
DeepSeek R1 Distill Llama 70B	70B	won't fit
Llama 3.3 70B Instruct	70B	won't fit
Qwen2.5 72B Instruct	72B	won't fit
Llama 3.2 90B Vision	90B	won't fit
Command R+	104B	won't fit
DBRX Instruct	132B	won't fit
Mixtral 8×22B Instruct	141B	won't fit
Qwen 3 235B (A22B)	235B	won't fit
DeepSeek Coder V2	236B	won't fit
Grok 2	300B	won't fit
Grok 1	314B	won't fit
Nemotron-4 340B Instruct	340B	won't fit
Jamba 1.5 Large	398B	won't fit
Llama 3.1 405B Instruct	405B	won't fit
DeepSeek V3	671B	won't fit
DeepSeek R1	671B	won't fit
Kimi K2 Instruct	1000B	won't fit