MoE Models Explained: Why Mixtral Uses 46B Parameters But Runs Like 13B

📚 More on this topic: Best Way to Run Qwen 3.6 35B MoE · A 177B Model on a 3060 · Mixtral VRAM Requirements · Our Benchmarks

Correction, 2026-09-10. From February to September this page said “below 48 GB, dense models win,” “MoE models are almost never the right choice” on a 24 GB card, and, under the decision framework, “Run dense models. Period.” Every MoE it named was Mixtral-era, its tier table returned Dense as the winner at every consumer size, and none of it knew about expert offload. Our own benchmarks contradict all of it. On our RTX 3090, Qwen3.6-35B-A3B loads whole at 21,652 MiB and decodes at 157.7 tok/s; Trinity-Mini does 206.6. The best dense model that fits the same card, Qwen3.6-27B at Q4, does 41.8, and 92.6 with speculative decoding bolted on. On a 12 GB 3060 the same 35B MoE runs at 38.9 tok/s with its experts in RAM, and a 177B MoE runs at 8.7 to 11.5 (stock and -ncmoe 41). The old rule was built from Mixtral’s shape and from VRAM arithmetic — with no measurement behind the comparison it turned on. Two things changed since Mixtral, each broke half of the rule, and the rest of this page is the rewrite. The routed-expert section further down was the one measured part of the old page and survives unchanged.

Mixtral 8x7B has 46.7 billion parameters but runs at the speed of a 13B model. For two years that sentence was the whole MoE story for anyone with a consumer GPU — and the moral was simple: the speed is real, the memory savings are not, so a 24 GB card should run a dense 32B and forget about MoE.

I wrote that moral down on this page and defended it for a year. Then I put an A3B-class MoE on my own 3090, watched it fit with 2.4 GiB to spare and decode nearly four times faster than the dense model I had been recommending instead — and had to work out what I had missed. The answer is two things — and neither is subtle once you see it.

This page still explains what a mixture of experts is and why Mixtral fooled people. It now also explains what changed, where the wall moved to, and the three cases where dense is still the right pick. Every speed on it is ours, measured on the two cards most of our readers own, with a link to where it came from.

What mixture of experts actually is

A standard “dense” model, like Llama 3.3 70B or Qwen3.6-27B, activates every parameter on every token. If it has 27 billion parameters, all 27 billion do work each time the model processes a word.

MoE takes a different approach. Instead of one massive feed-forward network (FFN) in each layer, an MoE model has several smaller FFNs called experts. A lightweight gating network, also called a router, looks at each incoming token and decides which experts should handle it. Only the selected experts run. The rest sit idle for that token.

Think of it as a hospital. A dense model is a single doctor who handles cardiology, neurology and orthopedics for every patient. An MoE model is a hospital with eight specialists. Each patient sees two of them — but the hospital still pays all eight salaries and keeps all eight offices.

The core components

Every MoE model has three parts:

1. Shared layers: attention, embeddings and layer norms that process every token, same as a dense model
2. Expert FFN blocks: several parallel feed-forward networks, only some of which run per token
3. Gating network: a small learned router that scores each expert for each token and picks the top-k

The gating network is tiny, typically a single linear layer per MoE block. It takes the hidden state of the current token, scores each expert, applies a softmax, and routes the token to the top-k experts. The selected experts’ outputs are combined using the gating scores as weights.

Token → Attention (shared) → Gating Network → [Expert 2, Expert 5] → Weighted Sum → Output
                                               (6 experts idle)

This routing happens independently for every token. One word might use experts 1 and 4, the next experts 3 and 7. Which experts specialise in what is learned during training — not designed.

Mixtral 8x7B: the model that set the rule

Mixtral 8x7B brought MoE to consumer hardware in late 2023, and its shape is the reason the old rule existed.

Eight large experts, two active. Mixtral computes about 13B parameters per token, so it runs at roughly the speed of a dense 13B. That was the win. The catch was memory: all 46.7B parameters have to be loaded, because the router decides at runtime which two experts each token needs, and at Q4 that is about 26 GB. It did not fit a 24 GB card. Meanwhile a dense 32B at Q4 took about 20 GB and did fit. The model that computed fewer parameters per token needed more memory — and the dense model beat it on quality anyway.

That is the VRAM trap — it was real, and it is still real for Mixtral. The mistake was reading it as a property of the architecture.

The two things that changed

The old rule had two halves: an MoE costs you the VRAM of its full parameter count, and at any VRAM budget a dense model of that size is better. Two developments broke one half each.

1. Fine-grained MoE with a small active count

Mixtral’s experts were huge, so its active count was 13B and its total was 47B — and the total was what killed it on a 24 GB card. The current generation goes the other way: many small experts, very few parameters per token, and a total that happens to fit.

Qwen3.6-35B-A3B has 256 experts per layer across 40 layers and routes each token through 8 of them plus one shared expert. That is 35B total and about 3B active. At UD-Q4_K_M it is a 20.6 GiB file, and on our RTX 3090 it loads whole, no offload flag at all, peaking at 21,652 MiB against the 24,123 MiB the card has free headless. Same footprint class as Mixtral at Q4 — and it fits where Mixtral did not.

Then it runs like a 3B model. Measured with llama-bench on that card, and every figure below is on our benchmarks page with its rig and flags:

Against the plain dense model that fits the same card, the two MoEs are 3.8x and 4.9x faster. Against dense’s best trick, DFlash speculative decoding, which needs a second model on the card and a fork of llama.cpp — they are still 1.7x and 2.2x. The Qwen3.6-27B figures come from the 3.8-vs-3.6 head-to-head and the DFlash bench; the 35B-A3B run is on its own page; Trinity was run on the same card for this page.

Nothing about the VRAM trap was wrong. Total parameters still set what you load. What changed is that a 35B total now fits a 24 GB card — and 3B active is what you pay per token. The first half of the old rule, “MoE needs more VRAM than you have,” stopped being true the day a well-shaped 35B shipped.

2. Expert offload moved the wall from VRAM to RAM

The second change is a flag. llama.cpp’s --n-cpu-moe N, -ncmoe for short, keeps the routed experts of N layers in system RAM and leaves attention and the shared weights on the GPU. A model whose experts overflow the card still runs; the experts cross the bus as each token selects them.

That flag changes what “total size” is a requirement of. With offload, total parameters set the RAM you need, and the GPU only has to hold the non-expert weights plus however many expert layers you can fit. The wall is still there — it moved from the card to the motherboard.

Qwen3.8-Flash-Next is 177B total, an 82 GB file at UD-IQ3_XXS, and no consumer card holds it. With -ncmoe 29 my 3090 with 62 GB of RAM decodes it at 22.6 tok/s at empty context and 24.5 with 4,096 tokens in place. On a 12 GB RTX 3060 with 32 GB of RAM it runs at 8.7 tok/s stock and 10.9 at 4K depth. Stock on the 3090 is 16.9, because stock puts nearly nothing on the card — the flag is worth 34 to 40 percent.

The 35B MoE tells the same story on the small card. On our 3060, Qwen3.6-35B-A3B with 24 of its 40 expert layers in RAM decodes at 38.9 tok/s, 38.2 at 8K depth, in 9.8 GB of VRAM. The dense model that fits a 3060 outright, Qwen3-14B at Q4, does 35.9 on the same box. A 35B model on a 12 GB card, faster than the 14B that fits it — that is the sentence the old page said could not exist.

When the experts don’t live in VRAM

Once the experts live in RAM the question stops being “does it fit” and becomes “how much has to cross the bus per token.” Our usual shorthand for that is active parameters, and against total parameters it is the right call. It is also coarser than it looks. Only part of a model’s active parameters are routed experts; attention, the shared expert and the dense layers stay on the GPU and never cross.

Volume is only half of it. The other half is whether the same experts keep coming back, because that decides whether any of the traffic is cacheable rather than merely unavoidable. We measured that directly for one model in what the router actually does in Qwen 3.6-35B-A3B.

The number you actually want is the routed-expert volume, and any GGUF header carries the fields to compute it:

routed params/token = 3 x embedding_length x expert_feed_forward_length
                        x expert_used_count x (number of MoE layers)

The 3 covers the gate, up and down matrices inside each expert. Count MoE layers from the tensor table rather than block_count, because some architectures put dense layers at the front that hold no experts at all. Three models we have measured:

Qwen and Trinity wear the same “3B active” label and differ by half in what moves. Trinity spends 50% of its active budget on routed experts against Qwen’s 34%, because its experts are twice as wide even though it has ten fewer MoE layers.

It shows up in the measurement. On our RTX 3090, pushing every expert out to RAM adds 28.98 ms per token on Trinity and 21.40 ms on Qwen. That is 1.35x, from two labels that predict no difference at all.

Be careful how much you read into that. The measured value lands between the two predictors in both pairs, and on Gemma-versus-Qwen active is the closer of the two (4% low against routed’s 7% high); what breaks it is the Trinity pair, where it predicts a dead tie and lands 26% low. Routed volume overshoots by 7-11% in both, so it is a rough guide and not a model of your tok/s. Active was falsified here. Routed was not confirmed. (Qwen’s offload sweep is on its own page; Trinity’s is a headless run on the same card, and the Flash-Next sweep walks -ncmoe layer by layer on both rigs, with the per-layer cost in a column.)

Be careful what it is a guide to, as well. Those figures are what offloading adds, not what you end up with, and the difference decides real comparisons. On a 12GB RTX 3060 with every expert in RAM, Trinity finishes ahead of Gemma at 23.70 tok/s against 21.20, despite moving more routed bytes per token. Routed volume ranks that pair backwards. What it leaves out is the work that never crosses the bus: Trinity’s non-expert weights are about 36% lighter than Gemma’s, and on a card that can hold neither model resident, that term is big enough to decide the race. Routed volume predicts the cost of offloading. It does not predict the throughput you finish with.

Total parameters still decide whether the model loads at all, and active parameters are still the right first cut against total size. The refinement sits one level below: when two MoE models advertise the same active count, the routed share tells you which will suffer more from being offloaded. It does not tell you which will be faster once both are.

Where dense still wins

Three cases, and I have a number for each — none of them is “below 24 GB.”

Quality per total parameter at equal file size. This is a judgement call, not a speed one, and it is the case the old page got half right. At 16.4 GiB, Qwen3.6-27B puts all 27B parameters into every token. At 20.6 GiB, Qwen3.6-35B-A3B puts about 3B in, drawn from a 35B pool. The MoE knows more and thinks less per token — the dense model is the reverse. On long reasoning chains and careful code review, 27B of work per token is worth something the 3B cannot buy back with speed, and it is why our Qwen 3.6 guide still sends coders to the 27B. We have not run the two head to head on a quality benchmark, so treat this as the site’s working judgement rather than a measurement, and treat “MoE has more total parameters, so it must be smarter” as exactly the kind of claim this page used to make in the other direction.

RAM too small to hold the expert set. Offload moved the wall to RAM — and RAM has a wall too. The 177B model’s expert set is about 1.5x a 32 GB machine, so on our 3060 box every decoded token pulls 25 MiB off the NVMe at 220 to 250 MiB/s, a 2,343-token prompt prefills at 31 tok/s, and there is no steady state to reach because the page cache never holds. That is 8.7 tok/s of chat and a long wait every time you paste a document. A dense model that fits the card has no such term. If the expert set does not fit in RAM with room to spare, the MoE is running off your SSD, and dense at the same VRAM wins on everything but breadth.

Prefill on offloaded MoE. Decode with experts in RAM is fine. Prefill is not, because a batch of prompt tokens touches every expert in every layer and the whole set crosses the bus. On the 3090, Qwen3.6-35B-A3B prefills at 3,674 tok/s resident and 412 with 32 of its 40 expert layers in RAM, a ninefold drop from the same flag that only cost decode 3.6x. The 177B model prefills at 137 tok/s on the 3090 with 62 GB of RAM. If your workload is long documents in and short answers out, an offloaded MoE spends most of its time in its slowest mode, and a dense model that fits the card — prefilling at 1,400 tok/s — will feel faster than its decode number suggests.

MoE models available today

The list this page used to carry was Mixtral, DBRX and DeepSeek V3. Here is the one that matters for a single consumer card.

The pattern to notice is the third column. Every model in the top block has a single-digit active count and hundreds of experts, and every one of them fits or offloads onto a card Mixtral could not touch. Mixtral’s problem was eight experts the size of a 7B model — not the architecture.

For Mixtral itself, the Mixtral VRAM Requirements page carries the per-quant tables for 8x7B and 8x22B; they are still correct for those two models and they are no longer on this page — because nobody choosing an MoE in 2026 should start from them.

What to run at each memory budget

This is the practical table. Every speed is our own llama-bench decode figure, empty context unless noted, from the article in the last column; the full rows with rig, flags and depth are on the benchmarks page.

Two things to read off it. At 24 GB the MoE column is not close — and it is not an offload story: both models are resident. And the 12 GB row is where the old page was most wrong, because “below 24 GB VRAM, skip MoE entirely” is falsified by a 35B model running faster than the 14B dense model that fits the card.

The 48 GB row is the one without a measurement — and I am not going to invent one. Neither of our rigs has that much VRAM.

How routing actually works

The gating network is simple but important. Here’s what happens at each MoE layer:

# Simplified pseudocode for MoE routing
def moe_forward(hidden_state, experts, gate):
    # Gate produces a score for each expert
    scores = gate(hidden_state)  # shape: [num_experts]

    # Softmax to normalize
    probs = softmax(scores)

    # Select top-k experts (k=2 for Mixtral, 8 for Qwen3.6-35B-A3B)
    top_k_indices = topk(probs, k=2)
    top_k_weights = probs[top_k_indices]

    # Renormalize weights so they sum to 1
    top_k_weights = top_k_weights / sum(top_k_weights)

    # Run token through selected experts and combine
    output = sum(weight * experts[idx](hidden_state)
                 for idx, weight in zip(top_k_indices, top_k_weights))

    return output

The gate itself is typically just a linear projection: gate_scores = W_gate @ hidden_state, where W_gate has shape [num_experts, hidden_dim]. For Mixtral with 8 experts and a hidden dimension of 4096, that’s an 8 x 4096 = 32,768 parameter matrix, negligible compared to the expert weights.

Load balancing

A naive gating network might route most tokens to the same two “favourite” experts, leaving the rest undertrained. To prevent this, MoE training includes an auxiliary load balancing loss that penalises uneven expert utilisation. The original Mixtral paper and the Switch Transformer paper cover this in detail.

In practice the load balancing loss does the job it targets: no expert sits idle, and every expert can handle general text. But “roughly evenly” does not survive a look inside a single layer, at least on a model carrying hundreds of experts. We traced every routing decision Qwen 3.6-35B-A3B makes — 256 experts per layer, eight picked per token — across six workloads, and each layer’s busiest tenth carries 42-55% of that layer’s work, four to five times uniform. Pool the same traces across all 40 layers and the distribution flattens to about 17%, which is where the widely repeated claim that MoE routing is flat comes from and why it misleads: expert #5 in layer 0 and expert #5 in layer 20 are different weights that never substitute for each other. Full measurement in what the router actually does.

Research on Mixtral does show experts leaning toward particular languages or domains, with no expert exclusively specialised. Our own cross-workload numbers cut against the strong reading of that: German technical prose shared 82% of its hot expert set with English chat. Whatever these experts specialise in, it is not cleanly language.

Why experts specialise at all

The experts aren’t pre-assigned roles. They start as identical random initialisations and differentiate during training through the gating network’s routing decisions and the load balancing loss, the same way neurons develop feature selectivity — emergent, not designed. You can’t grab one expert and use it as a standalone model. Each is trained in the context of the router and the shared attention layers.

Why the labs chose MoE, and why it now reaches you

MoE models learn more per training FLOP because the full parameter count sees data during training while each step only computes a fraction. DeepSeek V3 was reportedly trained for about $5.5M, cheap for 671B parameters — and the per-token compute saving is why DeepSeek, Google and Qwen serve MoE from their APIs.

The old page said MoE was “a good architecture stuck on the wrong hardware.” That was true when the only consumer-reachable MoE was Mixtral. It stopped being true when the labs shrank the active count to 3B and the tooling learned to leave experts in RAM. The same property that makes MoE cheap to serve — little compute per token — is what makes a 35B model decode at 157 tok/s on a used card.

The decision framework

Pick the row that matches your box, then the one question underneath it.

8 to 12 GB of VRAM

Run an A3B-class MoE with its experts in RAM. Trinity-Mini at -ncmoe 12 does 49.3 tok/s on our 3060; Qwen3.6-35B-A3B at -ncmoe 24 does 38.9, and either beats the 14B dense model that fits the card outright at 35.9. Check RAM first: the 35B model’s expert set wants 32 GB with room to spare, and the 177B model does not fit 32 GB at all — so at this size it runs off your SSD at 8.7 tok/s and prefills at 31. The 12 GB guide has the full ladder; the 16 GB guide is the same story with more headroom.

24 GB of VRAM

Run Qwen3.6-35B-A3B resident. It fits with 2.4 GiB spare and does 157.7 tok/s, 153 at 8K depth, no flags. Trinity-Mini is faster still at 206.6 if its quality suits your task. The dense pick, Qwen3.6-27B at Q4, is the right one when the job is long reasoning chains or careful code review and you want 27B of work on every token; you pay for that at 41.8 tok/s, or 92.6 with DFlash if you are willing to run a fork. “Run dense models, period” was this section’s old text — the measurement that replaced it is 157.7 against 41.8.

24 GB plus 64 GB of RAM

Everything above, plus the 177B class. Qwen3.8-Flash-Next at -ncmoe 29 decodes at 22.6 tok/s on the 3090 with 62 GB of RAM. The RAM is the purchase and the card is the flag: the same model on a 3090 with 32 GB would stream from disk exactly as our 3060 did. If you are choosing between a second GPU and 64 GB of RAM for MoE, the RAM is cheaper — and it is the one that moves the wall. Watch prefill: 137 tok/s on a 2,343-token prompt is a wait you will notice.

48 GB and up

We have not measured this tier, and the honest framing is that it buys headroom — not a different class of model: the 24 GB row at Q8 with long context, or Flash-Next with more of its 48 expert layers on the card than the 19 a 3090 holds. Treat the 24 GB + 64 GB row as your floor. When we bench a 48 GB configuration, the number goes here.

What to remember

MoE gives you the speed of a small model with the memory of a big one — and that is still true. What is no longer true is that the big one’s memory has to be VRAM, or that it will not fit. A 35B MoE fits a 24 GB card whole and runs at 157.7 tok/s where the dense 27B does 41.8. With expert offload, total size sets your RAM requirement and the card holds what it can, which is how a 177B model runs on a 3060.

Dense wins three fights: quality per token at equal file size, machines whose RAM cannot hold the expert set, and prompt-heavy workloads on an offloaded MoE. Pick dense for those reasons, with those numbers — and not because of a VRAM rule this page used to print.

If you have a single 24 GB GPU, start with Qwen3.6-35B-A3B. If you have 12 GB, start with the same model and -ncmoe 24. If you have 64 GB of RAM and want to see what 177B feels like, the Flash-Next page has the flag and the number.

Related guides

• Best Way to Run Qwen 3.6 35B MoE Locally: the 3090 and 3060 offload sweeps this page leans on
• A 177B Model on a 3060: The 32 GB Number Nobody Measured: expert offload at the far end, both rigs
• Qwen 3.6 MoE Routing, Measured: what the router actually does, layer by layer
• Mixtral 8x7B & 8x22B VRAM Requirements: the per-quant tables for the models that set the old rule
• Our Benchmarks: every figure on this page with its rig, flags and date