Hardware guide: running open-source AI models locally

The 60-second version

Weights at fp16 take 2 GB per billion parameters. At Q4_K_M quantization (the mainstream local default) that drops to about 0.6 GB per billion. KV cache adds roughly 20–50 % of that on top when you push context out to 32K+. Round up and give yourself a 20 % buffer for CUDA and runtime overhead.

• 8B at fp16 → 16 GB base + KV, needs a 24 GB card for real workloads.
• 32B at Q4_K_M → 19 GB base, comfortable on a 24 GB 4090 with modest context.
• 70B at Q4_K_M → 42 GB base, needs a 48 GB Apple Silicon budget or an H100.
• 405B at fp8 → 405 GB base, requires 8× H100 80 GB with tensor parallelism.

Hardware tiers & model picks

Running 70B on 24 GB VRAM

The most common question: can I run Llama 3.3 70B on my RTX 4090? Yes, at Q4_K_M with reduced context. Concretely:

1. Pull a pre-quantized GGUF: ollama pull llama3.3:70b defaults to Q4_K_M (~40 GB stored, ~24 GB VRAM at load).
2. Configure your inference server for 8K context to start. KV cache at 8K on 70B ≈ 4 GB; at 128K it explodes to ~60 GB.
3. Use llama.cpp / Ollama for pure GPU offload, or vLLM if you need production throughput. LM Studio wraps llama.cpp with a nicer UI.
4. Expect ~5–15 tokens/sec on a single 4090. Adding a second 24 GB card (e.g. a used 3090) can double throughput via tensor parallelism.

Apple Silicon specifics

MLX is the fastest inference path on M-series Macs; llama.cpp Metal is a close second. Unified memory is more flexible than GPU VRAM — a 64 GB M3 Max can dedicate ~48 GB to model weights while leaving the OS 16 GB. Practical picks:

• 16 GB: 3B–7B at Q4, single-user chat.
• 32 GB: 14B–32B at Q4 comfortably.
• 64 GB+: 70B at Q4 with 32K context, or 32B at fp16.
• 128 GB (M3 Ultra Mac Studio): 100B+ MoE models like Mixtral 8×22B at Q4.