Your GPU Is Mostly Waiting: The Next Step in Local AI Speed
Run a 27B model on an RTX 3090 and it writes about 44 tokens a second. (27B means 27 billion parameters, the numbers the model learned in training, also called its weights. A token is a word-piece, roughly three quarters of a word, and tokens per second, or tok/s, is the speed everyone quotes.) Feed it a 4,100-token prompt and it reads that at 1,540 tokens a second. Same card, same weights: reading is 35 times faster than writing.
That gap is the whole story of local AI speed right now. The card isn’t short of math. It’s short of trips to memory, and while it writes, it spends most of its time waiting for the next batch of weights to arrive. Every speedup that has actually worked on my hardware this year attacks that wait. The new ideas coming out of the labs attack it harder, but none of them are ready for your desk yet.
I’ll show the wait with numbers from my own bench box, then what beats it today. That includes a pre-registered re-test (predictions written down and published before running anything) of the claim that llama.cpp 1.0 is 55% faster. (llama.cpp is the open-source engine that most local AI tools, Ollama included, run on top of.) The claim turned out to be right about the speed and wrong about the version. The lab ideas come after that, and the buying advice comes last.
Why writing waits: one trip through memory per token
To write one token, the GPU has to read every weight in the model. A 27B model at Q4_K_M is a 17.1 GB file. (Q4_K_M is a quantization format: it squeezes each weight from 16 bits down to roughly 4 to 5, so the model fits on a home card, at a small cost in quality.) To write the next token it reads all of that again. To write 44 tokens a second, it moves something like 650 to 750 GB a second. (Not every byte gets read on every token: the embedding table, the lookup that turns each token into numbers, is read one row at a time. Hence a range, not a figure.) How fast a card can move bytes between its memory and its math units is its memory bandwidth, and the 3090’s is rated at 936 GB/s. So the card runs at around three quarters of its memory speed while its math units sit mostly idle.
Reading a prompt is different. That phase is called prefill; writing the answer, one token at a time, is decode. In prefill the GPU reads each weight once and applies it to hundreds of prompt tokens at a time. Same bytes moved, hundreds of times more work done per byte. That’s why prefill hits 1,540 tok/s on the same card.
Here’s what that looks like across models I’ve measured on a 3090. They’re all on the same card and all single-user, but they come from different articles and builds, so read them as the shape of the thing, not a league table:
| Model (RTX 3090, one user) | Weights read per token | Decode tok/s | Source |
|---|---|---|---|
| Qwen3.5-27B Q4_K_M, dense | ~17 GB | 43.3-44.0 | this test, 2026-11-20 |
| Qwen3.8-27B UD-Q4_K_XL, dense | ~16.7 GiB file | 42.1 | Bonsai 2 vs Qwen3.8 27B |
| Ternary Bonsai 2 27B, PQ2_0 | 6.7 GiB file | 77.6 | same article |
| Qwen3.6-35B-A3B, mixture of experts, 3B active | ~3B of 35B params | 156-184 | 35B MoE guide |
“Dense” means every weight is used for every token. UD-Q4_K_XL is Unsloth’s take on the same 4-bit squeeze (Unsloth is a team that publishes many of the popular model files), and PQ2_0 is Bonsai’s ternary format, where each weight is just -1, 0 or +1, about 2 bits apiece.
Speed tracks bytes per token, not parameter count. Bonsai has the same 27B parameters as Qwen and runs about 1.8x faster because its file is 2.5x smaller. The mixture-of-experts model (MoE for short) is split into many small “expert” blocks, and each token only passes through a few of them. It has more parameters than either and runs four times faster because each token only touches about 3B of them. The rule of thumb that falls out is blunt: decode speed ≈ memory bandwidth ÷ bytes read per token.
There are two ways around that. Read fewer bytes per token (smaller quants, ternary models, MoE). Or get more than one token out of each read. The second one is where the action is.
What already works, measured here
Guess ahead, check in one pass
Speculative decoding guesses several tokens cheaply, then has the big model check all the guesses in one pass and keep the ones it agrees with. Its answer comes out the same as without the guessing, just faster. That check costs about as much memory traffic as writing a single token, because the weights get read once either way. If two of four guesses are right, you got three tokens for the price of one read. I explained it in speculative decoding explained. The variants differ in where the guesses come from. MTP (multi-token prediction) is a small “guesser” built into some models, like Qwen3.5. It predicts the next few tokens, the main model checks them all in one pass, and it keeps the ones it agrees with. No extra download, no separate model. DFlash is a separate small helper model, trained to match one particular big model, that drafts a whole block of tokens at once. The oldest trick is a tiny draft model from the same family. It’s the easiest to set up and often the worst: the homebrewserver report found a 0.8B drafter made the 27B slower.
On my 3090, MTP took Qwen3.6-27B from 38.3 to 61.4-63.8 tok/s back in May (the 60 tok/s write-up). In its own test setup, DFlash took Qwen3.5-27B from about 33.7 to 83-93 tok/s, depending on the task (DFlash on both Qwens). Here’s the head-to-head: DFlash vs MTP.
llama.cpp 1.0 and the 55% claim, tested
When llama.cpp hit 1.0.0 on November 13, homebrewserver.club tested it on an RTX 3090. They found single-request speed unchanged. With MTP on, Qwen3.5-27B went from 45.1 to 68.5 tok/s (short prompt) and from 46.5 to 71.9 (long prompt): +52% and +55% against a build from November 8. (llama.cpp numbers every change it ships as a build; 1.0.0 is build 12359.) Draft acceptance, the share of guesses the big model keeps, was the same on both builds, so the guesses weren’t better. The checking was faster. They put that down to a new CUDA kernel, a piece of GPU code (CUDA is NVIDIA’s GPU programming platform), added in pull request #31533, “few-row MMQ for BS 2-24”. It speeds up the case where the model works through 2 to 24 tokens at once, and an MTP check is exactly that case.
Before re-running it, I checked the git history, and that’s where the version number came apart. PR #31533 merged on November 12 as build 12342. That’s inside the 0.9 series. The last 0.9 build, b12352, and v1.0.0, b12359, are seven commits apart, and none of those commits touches inference code: build-system tweaks, example clean-up, docs and two version bumps. So “1.0 vs 0.9” depends on which 0.9 you mean.
So I tested three builds, and wrote the predictions down before any of them was built:
- b12199, the report’s baseline;
- b12352, the last 0.9;
- v1.0.0.
Everything else followed the report. Same model file, checked byte for byte against its sha256 fingerprint. Same
prompts and token counts. A 32,768-token context window (how much text the model holds at once), one request at a
time (-c 32768 -np 1), and MTP guessing up to 6 tokens ahead (--spec-type draft-mtp --spec-draft-n-max 6).
Temperature 0, meaning it always picks the most likely token so runs repeat, and thinking off, meaning no hidden
reasoning before the answer. Three measured runs per cell, each after an unmeasured warm-up run, with the GPU cooled
to 50 °C first.
| Decode tok/s, mean (min-max) | MTP off, short | MTP off, long | MTP on, short | MTP on, long |
|---|---|---|---|---|
| b12199 (0.9, Nov 8) | 43.9 (43.9-43.9) | 43.3 (43.2-43.3) | 52.6 (51.9-53.9) | 58.5 (54.9-61.0) |
| b12352 (last 0.9) | 44.0 (44.0-44.0) | 43.4 (43.4-43.4) | 71.9 (71.2-73.2) | 78.3 (72.9-82.0) |
| v1.0.0 | 44.0 (44.0-44.0) | 43.4 (43.3-43.4) | 74.5 (73.2-75.5) | 75.0 (71.1-81.9) |
With MTP off, the version number does nothing. All three builds land within 0.1 tok/s of each other, so my first prediction (1.0 within ±3% of both 0.9 builds) held.
With MTP on, the jump is real, and it happened in 0.9. Going from b12199 to b12352 is worth 37% on the short prompt and 34% on the long one. From b12352 to 1.0, nothing that rises above the noise.
Against the report’s baseline, 1.0 comes out 42% faster on the short prompt and 28% faster on the long one. I’d predicted 1.30x to 1.70x at both lengths, so the long prompt missed by a hair, and a miss is a miss. My old build already got more out of MTP than theirs did (58.5 vs 46.5 tok/s on the long prompt), which left less room for the jump.
The third prediction missed too, and that one taught me something. I’d said 1.0 would land within ±3% of the last 0.9 with MTP on, since the code is identical. It came in at +3.7% and -4.2%. The per-request data shows why. With MTP, each request’s speed depends on how many guesses land, and that changes with the text: one request at 31% acceptance ran 71 tok/s, the next at 38% ran 82. Three runs per cell leaves MTP numbers wandering about ±7%. My band was too tight for that. It says nothing about two builds that share the same code.
Two more numbers worth knowing. Acceptance was the same in every build: about 30% of guesses on the short prompt, 33 to 36% on the long one. MTP costs VRAM, the card’s own memory: peak use went from 18.1 GB to 19.6 GB. Full numbers, logs and the void checks (the rules that throw out a measurement if something disturbed it) are in the record.
The bottom line: if you’re on a llama.cpp build from before November 12 and you use MTP on an NVIDIA card, update. It’s worth 28 to 42% on this model and card. Whether the build says 0.9 or 1.0 doesn’t matter.
What’s coming, unproven
Speculative decoding gets two to four tokens per weight read on a good day. Two research directions want a lot more than that. Both are real. Neither has shown it beats MTP on a home GPU.
Diffusion language models
A diffusion LM (language model) starts from a block of blank tokens and fills in many of them per pass, refining the whole block over a series of steps. It’s the same lever as MTP (more tokens per read of the weights) pulled much harder.
The headline numbers are big and come from datacenter hardware. Inception’s Mercury tech report cites independent measurements of 1,109 and 737 tok/s for its two coding models, on H100s, NVIDIA’s datacenter GPUs. Locally, llama.cpp’s diffusion example runs Dream, LLaDA and RND1 today. Google’s DiffusionGemma, a 26B mixture-of-experts, has GGUF files (the model format llama.cpp loads) and a llama.cpp pull request, PR #24423, that has been open since June.
The gotcha is in that PR’s comments. One RTX 5090 owner got 167-190 tok/s from DiffusionGemma, and 248 tok/s from regular Gemma 4 with MTP on the same card. On a 3090, posters report about 130 ms per diffusion step. Even the thread can’t agree on how to turn steps into tokens per second, so I’m not quoting a 3090 tok/s figure. Another commenter found step time climbing as a conversation grows: each step re-reads the whole context, without the KV cache reuse normal models get. (The KV cache is the model’s saved notes on text it has already read, so it doesn’t redo that work for every new token.) These are community numbers on an unmerged branch.
What to expect: on a consumer GPU, a diffusion step hits the compute limit that normal decoding never gets near. The spare math that makes MTP nearly free gets spent. Diffusion may still win for some workloads. Nobody has shown it winning on a 3090 against a tuned MTP setup.
Next-Concept Modeling (NCM)
NCM is a Stanford HazyResearch preprint from October 9 (arXiv 2610.14162). It compresses text into a few levels of codes, coarse to fine, and generates one whole level per pass, so a passage takes a handful of passes instead of one per token. The paper reports matching or beating diffusion LMs of the same size at 10 to 31 times the speed, and beating both diffusion and autoregressive models (the normal kind, writing one token at a time) on Sudoku.
I’m not getting excited yet. Every speed comparison in the paper is against diffusion models; there’s no timed comparison against a normal model, let alone one running MTP. The timings are at batch 16 (16 requests processed together) on an H100, which tells you little about one person on a consumer card. And the largest generator is 725M parameters, a few percent of the size people run at home. There’s research code and saved models for a few test tasks, but nothing you can chat with and no llama.cpp support.
The name invites a comparison with Meta’s “Large Concept Models”. They’re different ideas: Meta’s concepts are sentence embeddings (a list of numbers standing for a whole sentence), NCM’s are levels of codes.
What it means if you’re buying
Memory bandwidth still sets your floor. Without speculative tricks, decode ≈ bandwidth ÷ model size, and that’s what you get on every model that doesn’t ship an MTP head. I’d compare cards on GB/s first, VRAM second and teraflops (raw math speed) last. The 3090’s 936 GB/s is why it’s still the value card for local AI at about $1,500-1,625 used (eBay sold listings, November 6). The used 3090 buying guide covers what to check.
The tricks want headroom. MTP added 1.5 GB of VRAM here. A 27B at Q4 with MTP and a 32K context peaked at 19.6 GB. That fits 24 GB comfortably and doesn’t fit 16 GB at all. A drafter model needs its own memory too. If you’re choosing between a 16 GB card and a 24 GB card for 27B-class models, the speedups only show up on the 24 GB one.
Update your software before your hardware. On the same card, one CUDA kernel was worth 28-42% with MTP. The homebrewserver report says that kernel helps RTX 30-series and newer, and doesn’t help CPU, AMD or Apple. I’ve only tested it on a 3090.
And don’t buy for diffusion or NCM. Neither has a home-GPU result that beats what you can run tonight. If diffusion does take off locally, it will lean on compute more than today’s decoding does, which favours newer cards. That’s a guess, not a measurement, and it isn’t worth paying extra for now.
What I’ll test next
- DiffusionGemma against Gemma 4 with MTP, on the 3090, once PR #24423 merges. Same pre-registration rules, and an agreed way to count tokens per second.
- MTP with more runs per cell, so differences under 10% become measurable. The ±3% miss above says three runs aren’t enough when acceptance swings.
- The four-requests-at-once case. The report found +22% for the 27B on 1.0. I measured single requests only.
- NCM, if a checkpoint big enough to hold a conversation shows up.
Start with the cheap win: update llama.cpp, turn on MTP if your model has the head, and keep the VRAM for it. The next big step in local speed is coming from getting more tokens per trip to memory, and right now MTP is the version of that you can actually run.
A weekly email with every new guide and measured benchmark.
Subscribe — free, no spam