The Hitchhiker's Guide to the Galaxy notes that the most commonly uttered phrase in the known universe is not "hello" or "help" but rather "CUDA out of memory." It ranks just above "I thought 8GB would be enough" and just below "what do you mean the KV cache needs RAM too?"
Running an open-weight model isn't just about fitting the weights into VRAM. You need memory for the model, the KV cache, the inference framework overhead, and a safety margin. Here's how to size it properly.
§1 The Core Formula
Your total VRAM requirement is:
Total VRAM = Weights + KV Cache + Framework Overhead + Safety Margin
Let's break each piece down.
1. Model Weights
The weight memory depends on the number of parameters and the precision (bytes per parameter):
Weights (GB) = Parameters (billions) × Bytes per param ÷ 9.537
(Dividing by 9.537 converts billions of bytes to GiB — the unit your GPU reports.)
| Format | Bytes/Param | 7B Model | 70B Model | Notes |
|---|---|---|---|---|
| FP32 | 4.0 | ~28 GB | ~280 GB | Training only. Nobody serves in FP32. |
| FP16 / BF16 | 2.0 | ~14 GB | ~140 GB | Standard for unquantized inference. |
| FP8 | 1.0 | ~7 GB | ~70 GB | Hardware-dependent (H100, Ada). Slight quality loss. |
| NVFP4 | 0.5 | ~3.7 GB | ~37 GB | NVIDIA 4-bit float (Blackwell+). Better than INT4 at same size. Requires Hopper/Blackwell. |
| MXFP4 | 0.5 | ~3.7 GB | ~37 GB | OCP Microscaling 4-bit float. Compresses weights+activations. Calibrated variants available. |
| INT8 | 1.0 | ~7 GB | ~70 GB | Good quality/size tradeoff. Broadly supported. |
| INT4 / Q4 | 0.5 | ~3.7 GB | ~37 GB | The sweet spot for consumer GPUs. GGUF default. |
| Q2 / Q3 | 0.25–0.375 | ~1.8–2.6 GB | ~18–26 GB | Noticeable quality degradation. Desperation mode. |
| Ternary (1.58-bit) | ~0.22 | ~1.6 GB | ~15.5 GB | Weights stored as {−1, 0, +1} with FP16 group scaling (BitNet-style). Now a viable deployment format: Ternary Bonsai 2 27B keeps 98.2% of its FP16 parent at 1.76 bits/weight, where 2-bit quants lose ~16%. Needs a compatible runtime (llama.cpp PQ2_0/PTQ1_0, MLX, GPU kernels). |
MoE models: For Mixture-of-Experts models, weight memory is based on the total parameter count (all experts), not the active parameter count. A 2.4T MoE with 95B active still needs VRAM for all 2.4T parameters.
2. KV Cache
The KV cache stores attention keys and values for every token in the context window. It grows linearly with sequence length and is often the hidden killer.
KV Cache (GB) = 2 × layers × hidden_dim × context_len × bytes_per_param ÷ 1,073,741,824
Where:
- 2 — one for Keys, one for Values
- layers — number of transformer layers
- hidden_dim — head_dim × num_heads × num_kv_heads (for GQA models)
- context_len — max tokens you'll actually generate/context
- bytes_per_param — usually 2 (FP16), same as weight precision
For Multi-Head Attention (MHA): hidden_dim = head_dim × num_heads. For Grouped-Query Attention (GQA) or Multi-Query Attention (MQA), use num_kv_heads instead of num_heads — this can dramatically reduce KV cache size.
Hybrid-attention models: a newer class of open models — Qwen3.8-Flash-Next (Gated DeltaNet in 3 of every 4 layers), GLM-5.3-Flash (sparse + linear attention) — replaces most full-attention layers with linear/recurrent layers that keep a fixed-size state instead of a per-token KV cache. The formula above significantly overestimates their real memory use, because only the remaining sparse-attention layers grow linearly with context. Where vendors publish compression figures (GLM-5.3-Flash: 4.44× smaller cache than its dense equivalent), treat those as the correction factor.
Common mistake: Sizing for the model weights and forgetting the KV cache. A 14B model at 4K context might need 2GB of KV cache. At 32K context, that becomes 16GB — more than the weights themselves at INT4.
3. Framework Overhead
Inference frameworks (vLLM, llama.cpp, TGI, etc.) need memory for:
- Computation buffers (attention, MLP intermediates)
- Tokenization and detokenization
- Batch scheduling and request queuing
- CUDA context and driver (~500MB–1GB)
Rule of thumb: 1–2 GB for small models, 2–4 GB for large models. vLLM tends to be on the higher end due to its paged attention allocator.
4. Safety Margin
Always add 10–15% on top of your calculated total. Fragmentation, unexpected batch sizes, and framework version changes will eat into your headroom. A model that "just fits" on paper will OOM in production.
§2 Choosing a Quantization Format
The format you choose is the single biggest lever for reducing VRAM requirements:
| Format | Quality Loss | Best For | Tooling |
|---|---|---|---|
| FP16/BF16 | None | When you have VRAM to spare and need max quality | vLLM, TGI, HF Transformers |
| INT8 | Minimal (~1-2%) | Servers with enough VRAM to halve from FP16 | vLLM AWQ, bitsandbytes |
| FP8 | Minimal (~1-3%) | Hopper/Blackwell GPUs, models shipping native FP8 weights | vLLM, SGLang, TensorRT-LLM |
| NVFP4 | Minimal (~1-3%) | NVIDIA's 4-bit float format, better math than INT4 | vLLM, TensorRT-LLM (Hopper+) |
| MXFP4 | Small (~2-4%) | OCP microscaling format, compresses weights+activations | vLLM, SGLang (calibrated variants) |
| INT4 (AWQ/GPTQ) | Small (~2-4%) | Consumer GPUs, best quality/size tradeoff | vLLM, AutoAWQ, AutoGPTQ |
| Q4_K_M (GGUF) | Small (~2-4%) | CPU/GPU hybrid inference, llama.cpp | llama.cpp, Ollama |
| Q2_K (GGUF) | Noticeable (~5-8%) | Last resort when VRAM is tiny | llama.cpp, Ollama |
| Ternary / 1.58-bit | Small (~2%, trained-or-rotated variants) | 27B-class models under 6 GB; browser (WebGPU), phones | llama.cpp (PQ2_0/PTQ1_0), MLX, gemlite |
AWQ vs GPTQ vs GGUF: AWQ generally preserves quality best at INT4. GPTQ is faster to apply but slightly lower quality. GGUF is the only format that supports CPU offloading natively — essential for models larger than your VRAM.
QAT (Quantization-Aware Training) checkpoints: some labs now ship official
models that were trained to be quantized — Google's Gemma 4 family has
offered them since June 2026. A QAT w4a16 checkpoint typically gives up <1% quality
versus 2–4% for equivalent post-training quantization, and cuts memory ~66% from
BF16. If a QAT checkpoint exists for your model, prefer it over a third-party GGUF of the
same size — same bytes, better math inside.
§3 CPU Offloading
When a model doesn't fit entirely in VRAM, you can offload some layers to CPU RAM. The tradeoff:
- Pro: Run models larger than your GPU. A 70B model at Q4 fits in 40GB VRAM + ~2GB on CPU on a 48GB card.
- Con: CPU memory bandwidth (~50GB/s DDR5) is ~40× slower than GPU (~2000GB/s HBM). Offloaded layers are a massive bottleneck.
- Rule: Keep at least 80% of layers on GPU for acceptable performance. If you're offloading more than 20%, buy a bigger GPU or use a smaller model.
llama.cpp (and Ollama) support this natively with the -ngl (number of GPU layers) flag. vLLM does NOT support CPU offloading — it requires the full model in VRAM.
§4 Worked Examples
Let's size servers for real models from the Guide, using real architecture specs.
K2 Horizon 7B
Edge
Parameters7B dense
Layers32
Hidden dim4,096
Attention heads32
KV heads8 (GQA)
Head dim128
Context4,096 tokens
Q4 (GGUF, most common):
Weights: 7B × 0.5 bytes ÷ 9.537 = 3.7 GB
KV cache (GQA, 8 KV heads): 2 × 32 × (128 × 8) × 4096 × 2 bytes ÷ 1.07B = 0.5 GB
Framework overhead: ~1.5 GB
Safety margin (15%): ~0.9 GB
Total: ~6.6 GB → Runs on an 8GB GPU ✅
FP16 (full precision):
Weights: 7B × 2.0 ÷ 9.537 = 14.7 GB
KV cache: 0.5 GB
Overhead + margin: ~2.5 GB
Total: ~17.7 GB → Needs a 24GB GPU (RTX 4090/3090) ✅
Perfect edge model. Q4 on an 8GB card gives you ~2GB headroom for longer contexts. Even at FP16 it fits comfortably on a 24GB consumer GPU.
Qwen3.8-27B
Small
Parameters27B dense
Layers64
Hidden dim5,120
Attention heads40
KV heads8 (GQA)
Head dim128
Context8,192 tokens
Q4 (GGUF — the "runs at home" option):
Weights: 27B × 0.5 ÷ 9.537 = 14.2 GB
KV cache: 2 × 64 × (128 × 8) × 8192 × 2 ÷ 1.07B = 2.0 GB
Overhead + margin: ~2.8 GB
Total: ~19.0 GB → Fits on a 24GB GPU with ~5GB headroom ✅
FP16 (full precision):
Weights: 27B × 2.0 ÷ 9.537 = 56.6 GB
KV cache: 2.0 GB
Overhead + margin: ~8 GB
Total: ~66.6 GB → Needs 2× 48GB or 1× 80GB A100 ❗
Q4 with 32K context (max context):
Weights: 14.2 GB
KV cache: 2 × 64 × (128 × 8) × 32768 × 2 ÷ 1.07B = 8.0 GB
Overhead + margin: ~4 GB
Total: ~26.2 GB → Won't fit a 24GB card. Need 32GB+ or shorter context ⚠️
This is the model that catches people out. Fits at Q4 on a 24GB card for short contexts, but extending to 32K pushes you past the limit. The KV cache quadruples from 8K to 32K context.
GLM-5.3 (753B/40B MoE)
Frontier
Total parameters753B (MoE)
Active parameters40B per token
Layers94
KV heads16 (GQA)
Head dim128
Context8,192 tokens
FP8 (minimum viable for frontier MoE):
Weights: 753B × 1.0 ÷ 9.537 = 79.0 GB
KV cache: 2 × 94 × (128 × 16) × 8192 × 2 ÷ 1.07B = 5.8 GB
Overhead + margin: ~14 GB
Total: ~98.8 GB → Needs 2× 80GB (A100 80GB) with tensor parallelism ❗
INT4 (maximum compression):
Weights: 753B × 0.5 ÷ 9.537 = 39.5 GB
KV cache: 5.8 GB
Overhead + margin: ~7 GB
Total: ~52.3 GB → Fits on a single 80GB A100 with room for 32K+ context ✅
This is why INT4 exists. FP8 barely fits across two A100s. INT4 gives you the same model on a single card with headroom. The 40B active params mean inference speed is reasonable — you're loading 40B of compute per token, not 753B. But ALL 753B must reside in memory.
DeepSeek V4.1-Flash (552B MoE)
Frontier
Total parameters552B (MoE)
Active (prefill)8B
Active (decode)16B
Context8,192 tokens
LicenseMIT
FP8 (recommended for this model):
Weights: 552B × 1.0 ÷ 9.537 = 57.9 GB
KV cache (FP4 compressed): ~1.2 GB (novel architecture, ~437× smaller than standard)
Overhead + margin: ~10 GB
Total: ~69.1 GB → Single 80GB A100/H100 ✅
INT4:
Weights: 552B × 0.5 ÷ 9.537 = 28.9 GB
KV cache: ~1.2 GB
Overhead + margin: ~5 GB
Total: ~35.1 GB → Fits on a 48GB card (RTX 6000 Ada, A6000) ✅
The Causal Encoder-Decoder architecture with FP4 KV caching makes this model's memory footprint disproportionately small for its parameter count. The 8B prefill / 16B decode split means prompt processing is extremely fast. This is the most VRAM-efficient frontier model in the Guide.
Phi-4 14B
Edge
Parameters14B dense
Layers40
Hidden dim5,120
KV heads16 (GQA)
Head dim128
Context16,384 tokens
Q4_K_M (GGUF, standard for Phi-4):
Weights: 14B × 0.56 ÷ 9.537 = 8.2 GB
KV cache: 2 × 40 × (128 × 16) × 16384 × 2 ÷ 1.07B = 4.9 GB
Overhead + margin: ~2.5 GB
Total: ~15.6 GB → Fits on a 16GB GPU (RTX 4080, RTX 5000 Ada) but barely ⚠️
Q4 with 4K context (shorter):
Weights: 8.2 GB
KV cache: 2 × 40 × (128 × 16) × 4096 × 2 ÷ 1.07B = 1.2 GB
Overhead + margin: ~2 GB
Total: ~11.4 GB → Comfortable on 16GB ✅
The KV cache at 16K context is nearly as large as the weights. This is the classic trap — a "small" model with a long context window. If you don't need 16K context, don't allocate for it. Context length is a dial you can turn down.
§5 Quick Reference Cheat Sheet
| GPU | VRAM | Best Model (Q4) | Best Model (FP16) | Typical Use |
|---|---|---|---|---|
| RTX 4060 / 3060 | 8 GB | 7B, short ctx | 3B or smaller | Edge, personal |
| RTX 4070 / 3070 | 12 GB | 14B, 4K ctx | 7B, short ctx | Dev, personal |
| RTX 4080 / RTX 5000 Ada | 16 GB | 14B, 16K ctx | 7B, 8K ctx | Dev, light serving |
| RTX 4090 / 3090 | 24 GB | 27B, 8K ctx | 14B, 4K ctx | Serious dev, serving |
| RTX 6000 Ada / A6000 | 48 GB | 70B, 4K ctx | 27B, 8K ctx | Production serving |
| A100 / H100 | 80 GB | 700B+ MoE, 8K ctx | 70B, 16K ctx | Data centre, frontier |
| 2× A100 (TP) | 160 GB | 750B+ MoE, FP8 | 700B+ MoE, Q4 | Large-scale production |
Rules of Thumb
- Weights rule: Q4 params in GB ≈ param count in billions ÷ 2 (e.g., 70B ≈ 35GB)
- FP16 rule: params in GB ≈ param count in billions × 2 (e.g., 70B ≈ 140GB)
- KV cache rule: for GQA models at 4K context, add ~10–15% of weight size. At 32K, add ~80–100%.
- MoE rule: size for TOTAL params, not active. Inference speed depends on active params.
- Overhead rule: add 2–4 GB for framework + CUDA context, then 15% safety margin.
- Offload rule: if >20% of layers are on CPU, you need a bigger GPU or a smaller model.
Recommended Stack by Scale
- Single consumer GPU (8–24GB): Ollama or llama.cpp with GGUF files. Native CPU offload, easy to use.
- Single workstation GPU (24–48GB): vLLM with AWQ or GPTQ quantized models. Best throughput on single card.
- Data centre (80GB+): vLLM or TGI with tensor parallelism across multiple GPUs. FP8 or FP16.
- Broke but determined: llama.cpp with Q2_K and CPU offload. It'll be slow. Very slow. But it'll run.
Don't forget system RAM: Even with GPU inference, the model must be loaded into system RAM first before being copied to VRAM. You need at least as much system RAM as your total model size, ideally 2×. A 70B Q4 model (35GB) wants 64GB+ of system RAM.
Context length is a dial, not a fixed cost. Most frameworks let you set
max_model_len at load time. Setting it lower than the model's maximum context shrinks the KV cache allocation. If you're serving an API where most requests are under 2K tokens, don't allocate for 32K.
The Guide's final word on server sizing: if your calculations show you need exactly 23.8 GB and you have a 24 GB card, you do not have a 24 GB card. You have a 19 GB card after CUDA, framework overhead, and the inevitable memory fragmentation. Round up. Always round up. The universe is hostile to those who cut it fine, and so is the CUDA OOM error.