Ask why a model like DeepSeek-V3 can hold 671 billion parameters yet run at the cost of a model a fraction that size, and you arrive at the single most important architectural shift in modern large language models: the Mixture of Experts. MoE is now the default design for frontier systems, and understanding it explains a lot of otherwise-confusing behavior — why "parameter count" has stopped being a clean measure of cost, why some giant models feel cheap to serve, and why others are a nightmare to fine-tune.
This is a deep dive into how MoE actually works, the numbers behind the models using it, and the trade-offs that don't make it into the press releases.
The problem MoE solves
Start with a standard dense transformer. Every token you feed it passes through every parameter in every layer. A 70B dense model uses all 70 billion parameters to process the word "the," and again for the next token, and the next. Compute scales directly with size, which means making a dense model smarter by making it bigger makes it proportionally more expensive to both train and run.
That's the wall MoE is built to climb. The insight is simple to state: most tokens don't need most of the network. So why activate all of it every time?
How the architecture works
In a transformer, each layer has two main pieces: an attention block and a feed-forward network (FFN). MoE leaves attention mostly alone and replaces the single FFN with a set of smaller FFNs called experts, plus a small router (also called the gating network).
For each token, the router does something like this:
- It produces a score (a logit) for every expert.
- It applies a softmax to turn those scores into probabilities.
- It selects the top-k highest-scoring experts and sends the token only to them.
- It combines those experts' outputs, weighted by their routing scores.
This is sparse activation: out of, say, hundreds of experts, only a handful ever fire for a given token. The router is learned during training, so the model figures out its own division of labor — which experts specialize in what.
The vocabulary that matters here: a sparse MoE model is described by two very different numbers. Its total parameters determine how much memory the model occupies. Its active parameters determine how much compute each token costs. In a dense model these are the same number. In an MoE model they diverge — dramatically.
The numbers, from real models
Mixtral 8x7B (Mistral) is the clean textbook example. Each transformer layer has 8 experts, and the router uses top-2 routing — every token goes to exactly two of them. The result: 46.7B total parameters, but only ~12.9B active per token. Mistral's own framing is that it "processes input and generates output at the same speed and for the same cost as a 12.9B model," while achieving roughly a 72% compute reduction versus a dense Llama 2 70B at comparable quality. Note the counterintuitive naming — "8x7B" is not 56B, because the experts share the attention stack; only the FFNs are duplicated.
DeepSeek-V3 pushes the idea to the frontier. It carries 671B total parameters but activates only 37B per token — about 5.5% of the model doing 100% of the work on each step. Its MoE layers use a finer-grained design: 256 routed experts plus one always-on shared expert, with 8 routed experts selected per token. DeepSeek-V3 pairs this with Multi-head Latent Attention (MLA) and an auxiliary-loss-free load-balancing scheme — instead of adding a balancing penalty to the loss, it nudges a per-expert bias term during training to keep experts evenly used.
The pattern across the field is consistent, even where labs don't publish exact splits: MoE has become the dominant pattern for large models, reportedly underpinning systems from Google, Mistral, and others, precisely because it decouples capacity from cost.
A quick history
MoE isn't new. The sparsely-gated version that modern LLMs descend from was introduced by Shazeer and colleagues in 2017 with "Outrageously Large Neural Networks," which showed you could add enormous capacity to a network while keeping per-example compute roughly flat. Google's Switch Transformer (2021) simplified routing to the extreme with top-1 routing — each token goes to a single expert — and demonstrated the approach scaling to over a trillion parameters. Mixtral's top-2 routing sits deliberately between Switch's aggressive sparsity and a dense network, trading a bit of efficiency for quality.
The trade-offs nobody advertises
MoE is not a free lunch. It moves the difficulty around rather than removing it.
You still pay for total parameters in memory. DeepSeek-V3's 37B active parameters are cheap to compute, but all 671B must be loaded somewhere to be routable. That's why "you can run it locally" claims for giant MoE models come with heavy quantization and serious VRAM footnotes. MoE saves FLOPs, not storage.
Load balancing is a genuine engineering problem. If the router falls in love with a few experts and starves the rest, you've effectively trained a smaller model wearing a big model's clothes. The whole industry's move toward auxiliary losses and bias-based balancing (like DeepSeek's) exists because naive routing collapses.
Training can be unstable. Routing decisions are discrete-ish, gradients through the gate are finicky, and expert utilization can oscillate. MoE runs are more sensitive to hyperparameters than dense equivalents.
Distributed training gets communication-heavy. Experts are typically spread across devices, so routing a token to the right expert means shuffling activations across the network — the "all-to-all" communication that dominates the cost of large MoE training.
Fine-tuning is harder. The same sparsity that makes inference cheap makes adaptation awkward: you're tuning a routing behavior, not just weights, and small datasets can distort which experts get used.
The Bottom Line
Mixture of Experts is the reason "how big is the model?" stopped being a useful question on its own. The number that governs your bill is active parameters; the number that governs your hardware is total parameters — and MoE is the architecture that pried those two apart. It buys frontier-scale capacity at mid-size compute cost, which is why it now dominates. But it pays for that with memory footprint, routing complexity, training instability, and fine-tuning headaches. When you see a headline parameter count on a modern model, the honest first question is no longer "how big" but "how much of it actually fires."
