Nemotron 3.5 Bolt on One RTX 3090: 24 GB Holds at 128K

📚 More on this topic: What Can You Run on 24GB VRAM · KV Cache: Why Context Eats Your VRAM · The 35B MoE on a 3090 · Open Weights You Can’t Run · InsiderLLM Benchmarks

If you own a 24 GB card, long context is where the plan usually breaks. The weights take most of the card, and the KV cache grows with every token you keep. Nemotron 3.5 Bolt 74B-A9B 24GB is NVIDIA’s October 15 answer to that: a 74B model with 9B active per token, its experts quantized to IQ2_XXS at 2.0625 bits per weight, in a 22.02 GB file. NVIDIA’s model card says the architecture search “targets a deployment footprint below 24 GB at batch size 1 and 128K context.”

Then the same card’s recipe table starts at “RTX discrete GPU 32GB+”, with a 65,536-token context and a q8_0 KV cache. No 24 GB card is listed, and NVIDIA publishes no number for one.

So I measured it. One RTX 3090, llama.cpp v0.7.0, 32K and 128K context, f16 and q8_0 KV cache, plus a quality check on a task I already score other models on. I wrote my predictions down and locked them in a commit before anything ran. Five of the thirteen were wrong, and they’re all in the table below.

Does 24 GB hold? Yes, with room to spare

All four configurations loaded and served a 4,096-token prompt. Nothing ran out of memory, and nothing was re-fitted: every server ran with --fit off, so the context is what the command asked for.

Headroom is the card’s 24,576 MiB minus the peak nvidia-smi reading from server start through the first request. llama.cpp itself sees 24,122 MiB of usable device memory, so by its count the 128K f16 cell has 1,856 MiB free. Either way, there’s more than a gigabyte spare.

Going from 32K to 128K costs 864 MiB at f16 and changes short-prompt decode by nothing at all: 89.80 at both sizes. Then I filled the context. The table above is llama-server after a 4,096-token prompt; the depth figures from here on are llama-bench tg128 (128 generated tokens) on the same build, so compare them with each other, not with the table. With 102,400 tokens already in the cache, llama-bench decoded at 82.42 tok/s, which is 0.905 of the 91.07 it does at 4,096 tokens deep.

That’s the headline. A 74B model at 128K on one used 3090, still above 80 tok/s with about 100K of context behind it.

Why it fits: only 8 of 88 layers keep a KV cache

This is my reading of the architecture, from the GGUF’s own header and tensor list, not something NVIDIA’s card spells out in these terms. Bolt has 88 blocks. Forty are Mamba-2 layers, forty are MoE feed-forward layers, and only eight are attention layers, each with 2 KV heads of size 128.

Only those eight attention layers store anything per token. The arithmetic checks against the measured buffer: 8 layers × 2 heads × 128 × 2 (K and V) × 2 bytes × 131,072 tokens is exactly 1,024 MiB, and that’s the f16 KV buffer the server reported at 128K. The Mamba-2 layers keep a fixed-size state instead, and it measured 164.69 MiB at both 32K and 128K. It doesn’t grow with context.

Here’s where the memory goes at 128K, from the server’s own breakdown:

The weights are the whole story. If all 88 blocks were attention layers like those eight, the same arithmetic would give 11 times the cache: 11,264 MiB at 128K f16, more than the card has left. That’s also my read of why decode barely slows with depth: most of the network never reads the long context back. I haven’t profiled it layer by layer, so treat that as the likely reason, not a measured one.

q8_0 KV: fine at short context, slower when the cache is full

Normal advice for a tight card is to quantize the KV cache to q8_0 and halve its size. At short depth, q8_0 costs Bolt very little: 89.42 tok/s against f16’s 91.07 at 4,096 tokens deep, which is 1.8% slower.

Fill the context and that changes.

At 102,400 tokens deep, q8_0 decodes at 60.61 tok/s against f16’s 82.42. That’s 0.735 of the f16 speed, right when you’re using the context you bought the setting for. Each deep cell is a single repetition, because each one starts with a 102,400-token prefill, so read those two numbers as one measurement each.

The bottom line: on this model, use f16. The saving q8_0 buys is 479.99 MiB at 128K (a 544.01 MiB KV cache against 1,024.00), and Bolt at f16 already has 2,310 MiB spare. You’d be giving up 26.5% of your deep-context decode for memory you don’t need. On a model where the KV cache is the thing that doesn’t fit, the trade is different. Here it isn’t.

Quality: a tie with the control, and the control moved

Speed is half the question. The other half is whether a 2-bit 74B is any smarter than the 35B you’d otherwise run in the same card. I can only answer that on one task: the intent classifier from my own agent, three fields, exact match on all three, the frozen 47-item split I’ve scored since August (it’s the same split as the Qwen3.8 distill comparison). Temperature 0, thinking off, two passes per model.

A tie, and a close one item by item: 21 right for both, one that only Bolt got and one that only Qwen got.

Here’s the catch. The same Qwen file scored 23 / 47 on llama.cpp v0.4.0 in October. Same file, same prompts, same scorer; only the build changed, and five of its 47 answers changed with it. So a one-item swing on this split is within what a build change alone does. Read Bolt as “the same as Qwen3.6-35B-A3B on this task”, with ±1 as noise. That’s all this measures. It says nothing about coding, tool use, or anything NVIDIA’s own benchmark table covers.

Bolt took 0.488 s per item against Qwen’s 0.258 s on these short requests.

llama.cpp v0.7.0 vs v0.4.0: prompt speed up 14-16%, decode flat

Until this run, every mainline llama.cpp figure in our benchmark data came from v0.4.0 or an older build, and Bolt needs v0.7.0 (support landed in PR #30520, the day the model came out). So before putting a v0.7.0 figure next to an old one, I ran the same Qwen3.6-35B-A3B file, fully on the GPU, on both builds with the same command, alternating them three times.

Decode moved by under half a percent. It’s a real difference, outside the run-to-run spread, but you won’t feel it. Prompt processing went up 14-16%, which you will feel every time you paste a long document in.

It’s worth upgrading. Two flags will catch you out on the way. llama-server’s default port moved from 8080 to 9931 in v0.7.0, and --no-mmap / --mlock were removed in v0.4.1 in favour of --load-mode. I updated the reader commands in 15 guides for these on October 16, and the October 19 newsletter sums up both changes in one item. The CPU-only guide has --load-mode mlock in a full command. Old numbers on this site keep their build label, and new ones carry theirs.

--moe-cache-mib: +5% decode, and it cost prompt speed

v0.7.0 also adds --moe-cache-mib (PR #29887): when MoE experts live in system RAM, it keeps a cache of recently used ones in spare VRAM. The PR shows 1.62× decode on an RTX 4090 and 2.20× on a 5090. Our offload rows are the case it was built for, so I tested it on the same Qwen3.6-35B-A3B with 20 layers of experts in system RAM (-ncmoe 20), am17an’s nine-prompt set, a fresh server each run.

The cache bought +5.0% on decode and cost 5.6% on prompt speed. That’s a measurable gain, nowhere near 1.6×.

Two things to know before you read too much into the gap. First, the cache size. The PR says “10% of the total expert size is good,” and that’s the rule I locked in before running: 10% of this file’s 18,662 MiB of experts is 1,866 MiB, with 11,298 MiB free on the card. I tested that one size. A bigger cache might do better, and I haven’t measured one. Second, the PR’s headline rows aren’t at the 10% rule. Its 1.62× and 2.20× put every expert on the CPU (-cmoe) with 11,000 and 19,000 MiB caches, on a 93.7 GiB Qwen3.8-Flash-Next with a 16-core EPYC. Its own 10%-rule rows read 1.57× and 1.77×. That’s a different model, a different host, and a much bigger share of the model offloaded.

So the honest version is narrow: on a 3090 with a 35B-A3B at -ncmoe 20, the recommended cache size gives you 5%. I wouldn’t add it to a guide’s recommended flags on that.

What I predicted, and what happened

I wrote these down on October 16 and locked them in a commit before the card ran anything. Scored on the mean of the measured runs against the band I gave, without revision.

The misses mostly lean the same way: I expected Bolt to be tighter and slower at depth than it was, and I expected the cache to do more. The 128K miss is the good kind. I predicted it would barely squeeze in, and it had 2,310 MiB to spare.

How to run it

What I ran, in reader form. Point -m at your copy of the GGUF; on v0.7.0 the server listens on 9931 unless you pass --port.

llama-server -m NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf \
  -ngl 99 -fa on -c 131072 -np 1 -ctk f16 -ctv f16 --fit off -lm mmap

You don’t strictly need --fit off. With fit on, llama-server adjusts the arguments you didn’t set so the model fits in device memory. I turned it off so nothing was adjusted at all: a config that doesn’t fit fails instead, and you know. -np 1 keeps one slot, so the whole 131,072 is yours.

I measured text only. No --mmproj (the vision projector is another 1.61 GiB) and no speculative-decoding drafter (0.93 GiB). Add either one and you’re spending most of that 2,310 MiB of headroom, and I haven’t measured that.

Method and limits

• One card, one box. RTX 3090 24 GB at PCIe 4.0 x16, Ryzen 7 5700X, 64 GB DDR4-3000. Stock card, no gpu-unlock. Board-level power ran 380-407 W mean on the Bolt speed runs, against the card’s 420 W limit, and the SM clock gave up about 100 MHz to that cap. Nothing thermal.
• One build pair. v0.7.0 at commit 620d222 and v0.4.0 at 5266f24. Newer builds exist, including v0.8.0, and a later change may move these numbers.
• Text only. No vision projector, no drafter, so not NVIDIA’s own deployment.
• One quality task. My 47-item intent split, nothing else. A tie there is not a general quality claim.
• A display was attached. The HDMI port showed a connected monitor, with no desktop session; it held 7 MiB, which is inside every VRAM figure above.
• Short reps at depth. The 102,400-token cells are one repetition each.

Every figure here has its raw logs, commands, sha256s and the locked prediction file in the public record, including the deviations from the plan. The rows are in the benchmark dataset.

Bottom line

If you have a 24 GB card and want long context, Bolt loads at 128K on one 3090 with 2,310 MiB spare. Run it at f16 KV, with --fit off so you know the context you asked for is the one you got, and expect about 90 tok/s that stays above 80 with the context nearly full.

Don’t download it expecting a smarter model than Qwen3.6-35B-A3B. On the one task I can score, it tied. What you’re getting is the context window.

And update llama.cpp anyway: v0.7.0’s prompt speed is worth it on its own. Just check your port.

Related guides

• What Can You Run on 24GB VRAM?: where Bolt sits against everything else that fits the card
• KV Cache: Why Context Length Eats Your VRAM: the cache Bolt mostly does without
• Best Way to Run Qwen 3.6 35B MoE Locally: the control model, on this same card
• MoE Models Explained: why 74B-A9B stores like 74B and decodes like 9B
• Open Weights You Can’t Run: the other end of NVIDIA’s and everyone else’s release sheets