Best VRAM Cheat Sheet for Local LLMs: Every Model, Every Quant

📚 More on this topic: GPU Buying Guide · Qwen Models Family Guide · Quantization Explained · Context Length Explained · What Can You Run on 24GB VRAM · Qwen 3.5 9B Setup Guide · Llama 4 Guide

If you’re looking to run large language models locally, you’ve probably noticed that every guide eventually lands on the same question: how much VRAM do you actually need? The answer isn’t as simple as “more is better”—though that’s technically true. What matters is understanding the relationship between model size, quantization, and your specific use case.

This guide cuts through the confusion with concrete numbers based on real-world testing. You’ll learn exactly what fits in 8GB, 12GB, 16GB, and 24GB of VRAM, and which GPU makes the most sense for your budget.

Qwen 3.6 update — what changed for VRAM planning (April 2026)

Qwen 3.6 is the current-gen update, and it lands at two tiers: a 27B dense (released April 22, Apache 2.0) and a 35B-A3B MoE. Both carry the same VRAM footprints as their Qwen 3.5 equivalents. The 27B is ~17 GB at Q4_K_M — fits a 16GB card with light context, a 24GB card with room to spare. The 35B-A3B is 22.1 GB at UD-Q4_K_M (16.6 GB at UD-Q3_K_M) per the Unsloth GGUF card. What changed is the benchmark scores, not the memory math: the 27B hits SWE-bench Verified 77.2 per the Qwen 3.6-27B model card.

Qwen 3.6 currently updates only these two tiers. For everything else — the 0.8B through 9B small models, the 122B-A10B and 397B-A17B large MoEs — the Qwen 3.5 family table below remains the complete lineup reference.

The MoE offload trick (any MoE, not just 3.6). llama.cpp’s --cpu-moe flag parks the routed experts in system RAM and streams them over PCIe per token, keeping the shared expert and attention on GPU. A 16GB card plus 32GB system RAM runs the full Q4 of a 35B-A3B — Qwen 3.5 or 3.6 alike. RTX 5070 Ti 16GB and dual RTX 5060 Ti 16GB are viable hosts at 25–50 tok/s — and it goes lower: we measured a 35B-A3B at ~38 tok/s on a 12GB RTX 3060 (~10GB VRAM + ~11GB RAM). Slower than fully on-GPU, but it’s the full Q4 model, not an aggressive low-bit quant. If you were planning a 24GB card purely to run a 30B-class MoE, an MoE plus 32–64GB DDR5 is a real alternative. Full firsthand breakdown: Best Way to Run Qwen 3.6 35B MoE Locally.

Why VRAM Is the Bottleneck

The Memory Wall Problem

When you run an LLM, the entire model needs to be accessible for inference. Unlike gaming where textures can be streamed in and out, language models perform calculations across billions of parameters simultaneously. If those parameters don’t fit in VRAM, you’re stuck.

The math is straightforward: a model’s parameter count directly determines its base memory footprint. A 7 billion parameter model in FP16 (16-bit) precision uses approximately 14GB of VRAM. Double the parameters, double the VRAM. This is the memory wall for dense models, and no amount of clever optimization eliminates it — though Mixture-of-Experts models bend it by keeping only a few experts active per token and offloading the rest to system RAM, as the MoE section below covers.

Why System RAM and CPU Don’t Save You

You can technically offload model layers to system RAM, but the performance penalty is brutal. GPU memory bandwidth on an RTX 4090 hits 1,008 GB/s. Your DDR5 system RAM? Maybe 50-80 GB/s. That’s a 12-15x difference.

In practice, offloading a dense 70B model’s excess layers to RAM drops inference speed from 25+ tokens/second to 3-5 tokens/second. It works for testing, but it’s not a real solution for daily use — we measured the same collapse on a dense 14B: 35.9 tok/s fully on-GPU, 5.7 with half its layers in RAM, 3.05 all in RAM. A dense model runs every parameter every token, so every offloaded layer streams its full weights across PCIe each step.

But that’s a dense-model rule — and it’s the single thing this page used to get wrong for MoE. A Mixture-of-Experts model activates only a few experts per token, so offloading the idle experts to RAM barely costs anything: the slower RAM only has to feed a few hundred MB per token, not gigabytes. A 35B-A3B runs ~38 tok/s on a 12GB RTX 3060 with --n-cpu-moe (measured firsthand). So the corrected rule is: for MoE, active params plus an offload path — not total size — decide what you can run.

CPU-only inference is even worse. Running a 7B model on a modern CPU might give you 2-5 tokens/second compared to 40+ on a mid-range GPU. The CPU path exists for compatibility, not performance.

What Actually Lives in VRAM

VRAM doesn’t just hold model weights. You’re also paying for:

• Model weights: The actual parameters (the big one)
• KV cache: Stores attention state and grows linearly with context length
• Activation memory: Temporary tensors during forward pass
• Framework overhead: CUDA, your inference backend, typically 0.5-1GB

The KV cache is often overlooked. An 8B model at 32K context length needs approximately 4.5GB just for the KV cache with FP16 precision. At longer contexts, the KV cache can exceed the model weights themselves.

Qwen 3.5 partially sidesteps this problem. It uses Gated Deltanet (a form of linear attention) in 75% of its layers, which keeps KV cache growth much lower than standard transformer attention — especially at the 262K native context length these models support. This is one reason Qwen 3.5 9B fits so well on 8GB cards even with reasonable context windows.

VRAM Requirements by Model Size

7B-9B Models (Entry Point)

The 7-9B parameter range is where most local LLM journeys begin. Models like Llama 3.1 8B, Mistral 7B, and Qwen 2.5 7B pack solid quality into a small footprint. As of May 2026, Qwen 3.5 9B is the one to beat here. It outscores models 3x its size on reasoning benchmarks while fitting in 6.6GB on Ollama.

At Q4_K_M quantization, these models fit on 8GB cards with room for context. This is the entry point that actually works well.

Qwen 3.5 note: All Qwen 3.5 models are natively multimodal (text + images + video from the same weights). Processing images adds VRAM overhead on top of the base model — roughly 0.5-1.5GB depending on image resolution and count. If you’re using vision features on an 8GB card, stick to single images at default resolution to stay within budget.

13B Models (The Sweet Spot That Was)

The 13B class (Llama 2 13B, CodeLlama 13B) was the previous sweet spot before 7B models got smarter. They’re still relevant for specific fine-tuned variants.

A 13B at Q4 fits on 12GB cards. If you have 16GB, you can run Q6 or Q8 for better quality.

27B-34B Models (Serious Performance)

Models like DeepSeek-R1-Distill-Qwen-32B, Qwen 3.5 27B, and CodeLlama 34B are where you start seeing real jumps in reasoning and coding ability. You need real hardware for these.

Qwen 3.5 27B lands at ~17GB at Q4, which means it fits on a 24GB card with room for a reasonable context window. On Apple Silicon with 32GB+ unified memory, you can run it at Q6 or Q8 for better quality. An RTX 4090 or 3090 handles this whole range without issue. The RTX 5090 with 32GB is the first single consumer card that can run 32B models at Q8.

70B+ Models (The Big Leagues)

Llama 3.1 70B, Qwen 2.5 72B, and DeepSeek-V2.5 represent the upper limit of what’s remotely practical on consumer hardware.

Running 70B models requires either dual 24GB GPUs (48GB total), a single 48GB workstation card, or aggressive RAM offloading with significant speed penalties. Dual RTX 5090s (64GB total) can run 70B at higher quantization with approximately 27 tokens/second.

MoE Models: Big Brains, Smaller Footprint

Mixture-of-Experts (MoE) models change the VRAM math twice over. They have a large total parameter count but only activate a fraction of those parameters per token — and, crucially, the idle experts don’t have to sit in VRAM. With llama.cpp’s --n-cpu-moe you keep attention and the shared weights on the GPU and stream the routed experts from system RAM, so a big MoE runs on far less VRAM than its total size implies. A 35B-A3B needs ~20GB for its weights, but that can be ~10GB VRAM + ~11GB RAM — ~38 tok/s on a 12GB RTX 3060, measured. The “VRAM at Q4” figures in the table below assume the whole model sits in VRAM; treat them as the fully-on-GPU ceiling, not the floor.

The Qwen 3.5 family leans heavily on this architecture, and Llama 4 Scout joined the MoE party in April 2025:

The 35B-A3B is the interesting one for consumer hardware. Fully on-GPU it wants ~24GB at Q4, and inference speed is closer to a 3B model because that’s how many parameters actually compute per token — so on an RTX 3090 or 4090 it runs fast at 35B-level quality. But it does not require 24GB of VRAM: with --n-cpu-moe the experts move to system RAM and it runs on a 12GB RTX 3060 at ~38 tok/s (~10GB VRAM + ~11GB RAM), measured firsthand.

Llama 4 Scout sits in between — at ~55GB for Q4, it needs dual 24GB GPUs or an M5 Max with 128GB. It activates 17B parameters per token across 16 experts, so inference speed is closer to a 17B model. Aggressive quantization (Q2-Q3) can squeeze it onto a single 24GB card, but quality suffers.

The 122B-A10B needs an M4 Max or M5 Max with 128GB unified memory, or multi-GPU setups. The 397B is realistically cloud-only unless you have an M5 Max 128GB and are willing to run heavy quantization (Q3 or lower).

Qwen 3.5 Full Family VRAM Reference

The Qwen 3.5 family shipped in three waves (Feb 16 flagship, Feb 24 mid-range, Mar 2 small models). Every model uses a Gated DeltaNet hybrid architecture with 262K native context, and every one is natively multimodal (text + images + video from the same weights, early fusion, not bolted on).

The MoE nuance (corrected): The 35B-A3B only activates 3B parameters per token, so inference is fast — but all 35B still have to load somewhere (~20GB at Q4), so “3B active” does not mean “3B of memory.” The correction to the old rule: that ~20GB can be split across VRAM and system RAM. With --n-cpu-moe the idle experts stream from RAM, so the 35B-A3B runs on a 12GB card — not the 24GB the “Min VRAM” column implies — at ~38 tok/s on a 3060, measured. The same VRAM+RAM split lowers the floor for the 122B-A10B and 397B-A17B too.

KV-cache advantage: Qwen 3.5’s Gated DeltaNet architecture uses linear attention in 75% of its layers (3:1 ratio of DeltaNet to full attention). This keeps KV-cache growth significantly lower than standard transformers at long context lengths. A Qwen 3.5 9B running at 32K context uses roughly 40% less KV-cache memory than a standard 9B transformer would. This is part of why the 9B fits so comfortably on 8GB cards even with usable context windows.

How Quantization Changes Everything

What Quantization Actually Does

Quantization reduces the precision of model weights from 16-bit floating point to smaller representations. Instead of storing each parameter as a 16-bit number, you store it as 8-bit, 4-bit, or even 2-bit. For a deeper dive into how this works and which format to choose, see our quantization explainer.

The basic formula: VRAM (GB) ≈ (Parameters in Billions × Bits) / 8

A 7B model: FP16 = 14GB, Q8 = 7GB, Q4 = 3.5GB. Simple math, dramatic savings.

Common Quantization Levels

New Formats Worth Knowing (2026)

Two developments are changing the quantization landscape:

I-quants (IQ4_XS, IQ4_NL) use non-linear reconstruction with lookup tables instead of simple scaling. They retain ~95% quality at 4-bit — better than Q4_K_M — but decode slightly slower due to table lookups. If your hardware can handle the speed hit, IQ4_XS is the new quality king at 4-bit.

Unsloth Dynamic 2.0 GGUFs analyze each layer individually and pick the quantization type that minimizes accuracy loss for that specific layer. The result outperforms both imatrix and QAT-based quants on MMLU and KL divergence benchmarks. If you see a “Dynamic” GGUF on HuggingFace, it’s worth grabbing over the standard quant.

Quantize the KV cache, not just the weights. This is the lever most people miss. The KV cache is stored at FP16 by default, and at long context it can rival the weights for size (that 8B-at-32K example above is ~4.5GB of pure KV). You can quantize it too: q8_0 roughly halves KV memory with negligible quality loss, q4_0 roughly quarters it with a small measurable trade-off; vLLM’s --kv-cache-dtype fp8 does the same server-side. One gotcha to remember — in llama.cpp and Ollama, KV-cache quantization only takes effect when flash attention is also enabled (OLLAMA_FLASH_ATTENTION=1 alongside OLLAMA_KV_CACHE_TYPE=q8_0), otherwise the setting is silently ignored. On a tight card, quantizing the KV cache is often what buys you the context window you actually wanted.

Quality vs VRAM Tradeoffs

Q4_K_M is the sweet spot for most users. It’s the best tradeoff between quality and memory savings. Going lower (Q3, Q2) tanks quality and makes output unpredictable. Going higher (Q6, Q8) eats a lot more VRAM for diminishing returns.

For coding tasks, the quality gap between Q4 and Q8 is more noticeable. If you have the VRAM headroom, Q5_K_M or Q6_K makes a real difference for technical work.

Practical Recommendations by Use Case

Casual Chat and General Assistant

Minimum: 8GB VRAM | Recommended: 12-16GB VRAM

For everyday questions, summarization, and general conversation, a 7-9B model at Q4 quantization works well. Qwen 3.5 9B is the current best pick. Llama 3.1 8B and Mistral 7B Instruct are also solid and fit on 8GB cards.

If you want longer conversations without context window issues, 12GB gives you comfortable headroom for larger KV caches.

Best value: RTX 4060 Ti 16GB ($400) or used RTX 3060 12GB ($200)

Coding and Development

Minimum: 16GB VRAM | Recommended: 24GB VRAM

Coding tasks benefit from larger models. 7B models handle simple code completion fine, but 32B models like DeepSeek-Coder-V2 or CodeQwen are much better at understanding complex codebases.

At 24GB, you can run 32B coding models at Q4 with room for decent context windows. This is where the RTX 4090 and 3090 shine.

Best value: Used RTX 3090 (~$1,000) for 24GB at the best price-per-VRAM ratio — see our buying guide

Image Generation (Stable Diffusion, Flux)

Minimum: 8GB VRAM | Recommended: 12-16GB VRAM

Image generation has different VRAM characteristics than LLMs:

FLUX at full precision needs 22GB+, but NF4 quantized versions run on 6-8GB with minimal quality loss. For LoRA training, 24GB is strongly recommended.

Running Multiple Models / Hybrid Workflows

Minimum: 24GB VRAM | Recommended: 32GB+ VRAM

If you want to run an LLM and image generation simultaneously, or switch between multiple models without reloading, you need substantial headroom. The RTX 5090’s 32GB makes this practical for the first time on a single consumer card.

What You Can Actually Run: VRAM Tier Guide

8GB VRAM (Budget Entry)

Cards: RTX 4060, RTX 3070, RTX 3060 Ti

Qwen 3.5 9B is now the default recommendation for 8GB cards. At 6.6GB on Ollama, it leaves room for context and beats older 8B models on every benchmark. If you’re on 8GB, this is your model. See our setup guide for the walkthrough.

12GB VRAM (Practical Minimum)

Cards: RTX 4070, RTX 3060 12GB, RTX 3080 10GB/12GB

12GB is where local LLMs become actually useful. With Qwen 3.5 9B, you can run Q6_K (~9GB) or Q8_0 (~13GB, tight) for clearly better quality than Q4, especially on coding and reasoning tasks. You also get quality headroom on older 7-8B models and can dip into the 13B class. And the “27B+ won’t fit” rule of thumb is a dense-model rule: a 35B-A3B MoE runs here too, ~38 tok/s on a 12GB 3060 by streaming its experts from system RAM (measured).

16GB VRAM (Comfortable Middle Ground)

Cards: RTX 4060 Ti 16GB, RTX 4070 Ti Super, RTX 5060 Ti

The RTX 4070 Ti Super at 16GB is the performance choice here, hitting 25-35 tok/s. The RTX 4060 Ti 16GB is the budget choice at 12-18 tok/s — slower due to its 128-bit bus, but the VRAM capacity is the same. Qwen 3.5 27B at Q3 is technically possible (~14GB) but you’re leaving very little room for context. If you’re on 16GB, the 9B at Q8_0 is the smarter play.

24GB VRAM (The Sweet Spot)

Cards: RTX 4090, RTX 3090, RTX 5070 Ti

This is the serious enthusiast tier. Qwen 3.5 opens up two strong options here: the 27B dense model at Q4 (17GB, leaves 7GB for context) and the 35B-A3B MoE at Q4 (~22-24GB). The MoE variant delivers 35B-level quality but infers at roughly 3B speed since only 3B parameters fire per token.

Current-gen note: Qwen 3.6 shipped a 27B dense (~17GB Q4) and a 35B-A3B MoE (~24GB Q4) in April 2026 — same VRAM footprints as their 3.5 counterparts, with higher benchmark scores. On 24GB they’re the current equivalents of the Qwen 3.5 picks above.

The RTX 4090 (was $1,599 MSRP, now out of production and $2,300 used) delivers approximately 52 tok/s, while the used RTX 3090 ($1,000-1,300, median ~$1,150 after the 2026 memory-crunch run-up) hits around 42 tok/s. For pure VRAM-per-dollar, the 3090 is still unbeatable even after its 2026 appreciation.

48GB+ VRAM (No Compromises)

Cards: RTX 5090 (32GB), Dual 4090/3090 (48GB), RTX 6000 Ada (48GB)

The RTX 5090 at 32GB changed the math here, though the $1,999 MSRP is theoretical: street prices run $3,000-4,700. It runs 32B models at Q8 and can handle 70B at aggressive quantization on a single card. It hits 213 tok/s on 8B models and outperforms the A100 in many benchmarks.

For the Qwen 3.5 122B-A10B, you need 70-81GB at Q4. That means dual RTX 3090s (48GB, tight with Q3), an RTX 6000 Ada (48GB, same story), or Apple Silicon with 128GB unified memory. The M4 Max and M5 Max both support 128GB and are the most practical way to run this model locally. See the Apple Silicon section below.

Master VRAM Reference Table

Current-gen: Qwen 3.6-27B (~17GB Q4) and Qwen 3.6-35B-A3B (~24GB Q4) match the 24GB-tier footprints above; see the Qwen Models Family Guide.

→ Use our Planning Tool to check exact VRAM for your setup.

Apple Silicon: Unified Memory Changes the Math

Apple Silicon Macs share one pool of memory between CPU and GPU. There’s no separate VRAM — the entire unified memory pool is available for model weights. This means a MacBook Pro with 64GB of unified memory can load models that would require multiple discrete GPUs on a PC.

The tradeoff is bandwidth. Even the fastest Apple Silicon chips have lower memory bandwidth than a discrete GPU like the RTX 4090 (1,008 GB/s). You can fit bigger models, but they run somewhat slower per token.

Apple Silicon Memory & Bandwidth Comparison

The M5 generation is a big step up. The M5 Pro pushes 307 GB/s (up from ~200 on the M4 Pro), and the M5 Max hits 614 GB/s, which puts it closer to discrete GPU territory. Both chips also have Neural Accelerators baked into every GPU core, so the effective AI throughput is higher than the bandwidth numbers alone suggest.

What This Means in Practice

M5 Pro (64GB): Runs Qwen 3.5 27B at Q8_0 (~30GB) with room left for large context windows. The 35B-A3B MoE fits at Q4 (~22-24GB) and infers fast. This is a 32B-class workstation in a laptop.

M5 Max (128GB): The only consumer hardware that can load Qwen 3.5 122B-A10B at Q4 (~70-81GB) on a single machine without multi-GPU hacks. 70B dense models at Q8 fit with room to spare. At 614 GB/s, you’re looking at ~20-30 tok/s on a 70B model — not RTX 4090 fast, but fast enough for real work.

Upgrading from M4 to M5: Manage expectations on decode speed. The M4 Max already does 546 GB/s and the M5 Max does 614 GB/s — only about 12% more — and since token generation is bandwidth-bound, the raw tokens/sec gap on a 70B is smaller than the spec-sheet jump suggests. Where the M5 pulls ahead is prompt processing: the per-GPU-core Neural Accelerators make prefill (community reports cite roughly 3–4x faster time-to-first-token vs M4 Max), which you feel most on long-context and RAG work. If you already have an M4 Max 128GB, the same models fit; the upgrade buys you snappier prefill, not dramatically faster generation. If you have an M4 Pro 48GB, the M5 Pro’s jump to 64GB and 307 GB/s opens up models that flat out didn’t fit before.

For detailed Mac setup instructions, see our running LLMs on Mac M-series guide.

Non-Apple unified memory: DGX Spark and Ryzen AI Max

Apple no longer has this niche to itself. Two x86-adjacent boxes now offer 128GB of unified memory around the $4,000 mark, and both fit big MoEs a 24GB card can’t touch:

• NVIDIA DGX Spark (GB10) — 128GB LPDDR5x unified, CUDA-native. Great for fitting models; the catch is ~273 GB/s bandwidth, so generation on anything past ~20B is slow (single-digit to low-teens tok/s on 30–70B).
• AMD Ryzen AI Max+ 395 (Strix Halo) — 128GB LPDDR5x, Radeon 8060S iGPU, runs Windows or Linux. Similar story: the memory lets you load a 100B-class MoE; bandwidth caps how fast it generates.

The pattern to internalize: these boxes trade GPU-class bandwidth for capacity. They shine on MoE models where only a few billion params fire per token (so the bandwidth hit is smaller), and struggle on dense 70B where every parameter has to move each token. Full field comparison in the GB10 boxes guide.

The Bottom Line

If you’re buying new today (note: a 2026 memory shortage has pushed street prices well over MSRP across the board):

• Budget (~$430+): RTX 5060 Ti 16GB or 4060 Ti 16GB—slow but 16GB
• Mid-range (~$1,000+): RTX 5070 Ti 16GB—good balance, inflated by the shortage
• High-end: RTX 4090 24GB—out of production now, ~$2,300 used
• Flagship ($1,999 MSRP, ~$3,000 street): RTX 5090 32GB—if you can find one near MSRP, grab it

If you’re buying used:

• Best value: RTX 3090 at ~$1,000-1,300 (median ~$1,150) on eBay or Amazon—24GB at the best price-per-VRAM ratio despite the 2026 run-up

The minimum for a useful local LLM setup is still 12GB. But 8GB is more capable than it used to be, thanks to Qwen 3.5 9B fitting in 6.6GB. At 24GB, the Qwen 3.5 35B-A3B MoE gives you 35B-class quality at 3B inference speed. And on Apple Silicon, the M5 Max with 128GB at 614 GB/s runs models that needed a datacenter eighteen months ago.

VRAM is the one spec you can’t fake or work around. Buy as much as you can reasonably afford. For specific card recommendations, check our GPU buying guide.

Related Guides

• GPU Buying Guide for Local AI
• What Quantization Actually Means (And Why It Matters)
• Used RTX 3090 Buying Guide for Local AI
• Qwen 3.5 9B Setup Guide
• Qwen 3.5 Small Models: 9B Beats Last-Gen 30B
• Llama 4 Guide: Running Scout and Maverick Locally
• Running LLMs on Mac M-Series