What Can You Actually Run on 4GB VRAM?

📚 More on this topic: Best Models Under 3B Parameters · CPU-Only LLMs · 8GB VRAM Guide

Let’s be direct: 4GB of VRAM is not a lot. It was entry-level five years ago, and it’s the absolute floor for local AI today. But “floor” doesn’t mean “useless.” If you’ve got a GTX 1050 Ti sitting in an old PC or a GTX 1650 in a gaming laptop, you can do more than you’d expect — as long as you pick the right models and don’t try to punch above your weight class.

This guide covers exactly what fits, what runs well, what barely works, and when you’re better off going CPU-only or upgrading. No sugarcoating.

Which GPUs Have 4GB?

The AMD cards have 2x the memory bandwidth of the NVIDIA cards, which theoretically means faster token generation. But Ollama doesn’t support Vulkan, so you’ll need to use llama.cpp directly. If that’s not your thing, stick with NVIDIA — Ollama just works with CUDA.

What Actually Fits in 4GB

Here’s the math. Your 4GB needs to hold model weights + KV cache + ~500MB of runtime overhead. That leaves roughly 3.5GB for the model and its context.

The sweet spot for 4GB: Qwen3 4B at Q4_K_M. It’s ~2.5GB, leaves room for KV cache and overhead if you keep context tight, and it’s the successor to the old Qwen 2.5 3B pick with better reasoning. If you want more context headroom, drop to Qwen3 1.7B — it’s ~1.1GB and leaves plenty of room. Gemma 3 4B and Phi-4-mini are fine alternatives, just tighter.

The trap: 7B at Q2. Yes, a Q2_K 7B model fits in ~3GB. But Q2 quality is severely degraded — outputs are incoherent more often than not. Multiple benchmarks show a 4B model at Q4 outperforms a 7B at Q2. Don’t bother.

→ Check what fits your hardware with our Planning Tool.

Real-World Performance

These speeds come from benchmarks on the Quadro P1000 (4GB, comparable to GTX 1050 Ti class) and bandwidth-based estimates for the GTX 1650:

Everything above 15 tok/s feels responsive for chat. On these cards you’re comfortable up to ~1.7B; the 4B models run but sit right at the responsive threshold, and they tighten fast once you add context.

What Works Well on 4GB

These use cases are genuinely practical on a 4GB GPU:

Chat and Q&A (4B models): Qwen3 4B and Llama 3.2 3B handle general conversation, question answering, and brainstorming well. You won’t mistake it for a frontier model, but for quick local tasks it’s solid.

Simple coding assistance: Qwen3 4B handles code completion, explaining functions, and writing short scripts. Not full-project coding, but useful for autocomplete and quick snippets.

Text classification and extraction: Small models excel at structured tasks like sentiment analysis, entity extraction, and categorization. A Qwen3 1.7B at Q8 runs at 30+ tok/s and handles these well.

Summarization (short docs): Feed in a few paragraphs and get a summary. Keep your context window short — 2-4K tokens is the practical limit on 4GB with a 4B model.

Embeddings for RAG: Embedding models like nomic-embed-text (270MB) or all-minilm (80MB) fit easily. You can run embeddings on GPU while keeping the LLM on CPU if needed. See our RAG guide for setup.

What Doesn’t Work on 4GB

Be honest with yourself about these limitations:

7B+ dense models at usable quality: You can’t run Qwen3 8B, Llama 3.1 8B, or Mistral 7B at Q4 or above — they simply don’t fit. Q2/Q3 fits but the quality is so degraded it’s not worth using. Go CPU-only instead.

Big MoE models via expert offload: On a 12GB or 16GB card you can run a 35B-A3B Mixture-of-Experts model by streaming its idle experts to system RAM with llama.cpp’s --n-cpu-moe. That trick does not reach 4GB — more on why in the offloading section below. At 4GB it’s small dense models only.

Image generation (mostly): SDXL needs 6GB+ and won’t load. Flux is out of the question. SD 1.5 technically works at 512x512 with --medvram flags, but you’re looking at 30-80 seconds per image on a GTX 1050 Ti. It’s possible, but painful. See our Stable Diffusion guide for details.

Long context: With a 3B model at Q4 taking ~2GB of VRAM, you have about 1-1.5GB left for KV cache. That’s roughly 2-4K tokens of context with FP16 KV cache. You can stretch to 6-8K tokens with Q8 KV cache quantization, but forget about 16K+ conversations.

Running model + other GPU tasks: If your GPU is also driving your display, doing video decode, or running a game — subtract 200-500MB from your available VRAM. On a 4GB card, that can be the difference between a model loading or not.

GPU Offloading: The Partial Solution

If you want to try a 7B model, you can split it between GPU and CPU. Put some layers in VRAM (fast) and the rest in system RAM (slow). This is called partial offloading.

In Ollama

# Set number of layers on GPU (lower = more on CPU)
# A 7B model has ~32-36 layers
OLLAMA_NUM_GPU=10 ollama run llama3.1:8b

Or in a Modelfile:

FROM llama3.1:8b
PARAMETER num_gpu 10
PARAMETER num_ctx 2048

In llama.cpp

./llama-cli -m model-Q4_K_M.gguf -ngl 10 -c 2048 -p "Hello"

The -ngl flag controls how many layers go to the GPU. Start low and increase until you’re near the VRAM limit.

Is Partial Offloading Worth It?

Honest answer: usually not on 4GB.

Partial offloading on a 4GB card is often slower than just running on a decent CPU. The PCIe bus becomes the bottleneck — data constantly shuttles between VRAM and system RAM at 16 GB/s (PCIe 3.0), while your VRAM bandwidth is 112-256 GB/s.

When partial offload helps: If your CPU is old or slow (pre-Ryzen, low RAM bandwidth), even 10 layers on GPU can give a meaningful speedup. But if you have a modern Ryzen or Intel with DDR5, CPU-only is often faster for 7B models.

What About MoE Models and --n-cpu-moe?

You may have read that a Mixture-of-Experts model like Qwen3-30B-A3B runs on a 12GB card by streaming its idle experts to system RAM with llama.cpp’s --n-cpu-moe. That’s real, and it’s genuinely impressive — a 35B-A3B holds ~38 tok/s on a 12GB RTX 3060 because only ~3B parameters fire per token, so the RAM-resident experts barely cost speed. The full rule is in the VRAM requirements MoE section.

Here’s why it doesn’t help at 4GB: expert offload only moves the idle experts to RAM. The attention layers, the shared weights, and the KV cache still have to sit in VRAM — and for a 35B-A3B that on-GPU portion is roughly 10GB. A 12GB card has room for it; a 4GB card isn’t close. There’s no version of the trick that squeezes the resident part of a big MoE into 4GB.

So the honest rule for this tier: 4GB is small-dense-only. MoE expert offload is a real reason to want a 12GB card, not something you can reach from here — don’t let a benchmark from a 12GB card set your 4GB expectations.

Should You Upgrade or Go CPU-Only?

This is the real question. Here are your options:

Option 1: Go CPU-Only (Free)

If you have 16-32GB of RAM, CPU-only inference is a legitimate option:

• 7B models at Q4: 7-15 tok/s on a modern CPU
• 13B models at Q4: 3-7 tok/s with 32GB RAM
• No VRAM limitations — context length only limited by RAM
• DDR5 systems get noticeably better speeds than DDR4

Choose CPU-only if: You want to run 7B+ models and don’t want to spend money. Your 4GB GPU is still faster for 1-3B models, so use GPU for small models and CPU for larger ones.

Option 2: Upgrade the GPU ($120-200)

The used GPU market has great options that make 4GB look like a different world:

The RTX 3060 12GB at $200-400 (figure on about $300 since the 2026 memory shortage lifted used prices) is still the most recommended budget LLM card in the community. At ~$25 per GB of VRAM it opens up models that are genuinely competitive with cloud AI for everyday tasks — and, unlike your 4GB card, it’s the tier where the MoE offload trick starts to work.

Choose upgrading if: You want 7B+ models at full speed, plan to do image generation, or want headroom to grow.

The Honest Take

4GB VRAM is the floor, not a foundation. It’s enough to experiment with small models, learn the tools, and figure out if local AI is for you. But if you get hooked — and you probably will — you’ll want more VRAM within a week. Budget $150-200 for a used GPU upgrade and consider it an investment. The jump from 4GB to 12GB is the biggest quality-of-life improvement in local AI.

Getting the Most Out of 4GB

If you’re sticking with 4GB for now, these tips help:

Set context length explicitly. Ollama and llama.cpp default to large context windows that will OOM on 4GB. Always set num_ctx to 2048 or 4096 max.

# Ollama
OLLAMA_NUM_CTX=2048 ollama run qwen3:4b

# llama.cpp
./llama-cli -m model.gguf -ngl 99 -c 2048

Enable KV cache quantization. This halves the memory used by context:

# Ollama
export OLLAMA_KV_CACHE_TYPE=q8_0
ollama serve

# llama.cpp
./llama-cli -m model.gguf -ngl 99 -c 4096 -ctk q8_0 -ctv q8_0

With Q8 KV cache, a 3B model at Q4 can handle 4-6K tokens of context instead of 2-3K. No meaningful quality loss.

Try Flash Attention. On the GTX 1650 (Turing), Flash Attention gives a modest ~10% speedup on prompt processing and reduces VRAM usage. On the GTX 1050 Ti (Pascal), it saves VRAM but may not improve speed. Worth enabling either way:

# llama.cpp
./llama-cli -m model.gguf -ngl 99 --flash-attn -c 2048

Pick models with small KV caches. Qwen3 uses grouped-query attention with few KV heads, giving it a much smaller KV cache than Llama 3.2 or Phi-4-mini at a similar parameter count. Approximate KV cache at 4K context:

Qwen3’s KV cache is a fraction of Llama 3.2’s at similar size. On a 4GB card, that’s the difference between comfortable context and constant out-of-memory errors. Qwen3 is the best model family for 4GB VRAM — small dense sizes, strong quality per parameter, and a tiny KV footprint.

Close everything else. Your browser, Discord, even your desktop compositor uses VRAM. On a 4GB card, 200MB matters. Close what you can before loading a model.

The Bottom Line

4GB VRAM is enough to get started with local AI. Run Qwen3 4B at Q4, learn Ollama, try RAG with small models, experiment with what’s possible. You’ll get responsive chat, decent summarization, and basic coding help — as long as you stay in the small-dense lane and don’t expect the big-MoE tricks that need a bigger card.

But don’t fight the hardware. If you need 7B+ dense models or a 30B-class MoE, go CPU-only or buy a used GPU. The RTX 3060 12GB at $200-400 transforms the experience from “making it work” to “this is actually good” — and it’s the tier where expert offload finally runs a 35B-A3B. The used RTX 3090 at $1,000-1,400 opens up 24GB of possibilities that make 4GB feel like a different era.

Start small, learn the tools, upgrade when you’re ready.