Mamba: The State Space Models Challenging the Transformer
For seven years, "AI model" has meant "Transformer." Every frontier LLM you can name is built on the same attention mechanism from a 2017 paper. Mamba is the most serious attempt yet to change that — and in 2026, it stopped being a research curiosity and started shipping inside production models from NVIDIA, Alibaba, and AI21.
This is a technical deep dive into why state space models exist, what Mamba actually changed, and why the frontier is quietly converging on hybrids rather than picking a winner.
The problem Mamba is trying to solve
The Transformer's superpower is self-attention: every token can look at every other token. That's also its curse. Attention scales quadratically with sequence length. Double the context and you roughly quadruple the compute and memory for the attention step.
That quadratic wall is why long context is expensive, why serving models is memory-hungry, and why the KV cache balloons as conversations grow. For a decade, researchers chased subquadratic alternatives — linear attention, gated convolutions, recurrent models, and structured state space models (SSMs). They were faster. They just weren't as good at language.
The core weakness, as the Mamba authors identified it, was an inability to do content-based reasoning — to selectively decide what matters based on the actual input.
What a state space model actually is
An SSM borrows an idea from classical control theory. Instead of comparing every token to every other token, it maintains a compressed hidden state — a fixed-size summary of everything seen so far — and updates that state as each new token arrives.
Formally, an SSM maps an input sequence through a latent state using a recurrence: the new state is a function of the previous state and the current input, and the output is a function of the state. Crucially, that state is a fixed size. It does not grow with the sequence.
This is the whole game. Because the state doesn't grow:
- Compute scales linearly. Processing a sequence twice as long costs about twice as much — not four times.
- Memory stays flat. There's no ever-expanding KV cache. The state is a constant-size tensor.
- Inference is fast. Each new token is a cheap state update, not a scan over the entire history.
The catch, historically: a fixed recurrence applies the same transformation to every token. It can't decide that this token matters and that one is filler. That rigidity is exactly why older SSMs lost to attention on language.
Mamba's key insight: make the SSM selective
The Mamba paper — Linear-Time Sequence Modeling with Selective State Spaces by Albert Gu and Tri Dao, published in December 2023 — fixed the content-reasoning problem with one deceptively simple move.
Let the SSM's parameters be functions of the input.
In a classic SSM, the matrices that govern how the state updates are fixed. Mamba makes them input-dependent, so at each step the model can decide, based on the current token, whether to propagate information into the state or forget it. This is the selective in "selective state space model." Suddenly the model can focus, ignore, and route information the way attention does — while keeping the linear-time structure.
There was a cost. Input-dependent parameters break the efficient convolution trick that made older SSMs fast on GPUs. Gu and Dao answered with a hardware-aware parallel algorithm — a scan that keeps the expensive intermediate state in fast SRAM instead of shuttling it to slower GPU memory. Without that engineering, selective SSMs would be a nice theory and a slow implementation.
The final architecture is radical in its simplicity: Mamba drops attention and the MLP blocks that every Transformer relies on, folding everything into a single, homogeneous selective-SSM block.
The results that got everyone's attention
The headline numbers from the paper are what moved the field:
- 5× higher inference throughput than Transformers of comparable size.
- Linear scaling in sequence length, with quality that keeps improving on real data out to million-length sequences.
- A Mamba-3B model that outperforms Transformers of the same size and matches Transformers twice its size, in both pretraining and downstream evaluation.
State-of-the-art results across language, audio, and genomics from an architecture with no attention at all — that's what turned skeptics into investigators.
Mamba-2 and the great reconciliation
If Mamba was a challenge to the Transformer, Mamba-2 (2024) was a peace treaty. Its central contribution was theoretical: it established a formal duality between selective SSMs and attention, showing that the two can be viewed as different points on a single spectrum of sequence models rather than rival paradigms.
This structured state space duality mattered more than any benchmark. It meant SSM and attention layers could be reasoned about with shared math — and, more practically, mixed freely in the same network. That insight is the foundation for everything happening in 2026.
2026: the hybrids won
Here's the plot twist. The interesting question in 2026 is no longer "Mamba or Transformer." It's "how much of each, and where?" Three developments tell the story.
NVIDIA's Nemotron-H family (reported at 8B, 47B, and 56B) replaces roughly 92% of attention layers with Mamba-2 blocks, keeping a thin layer of attention where global lookups genuinely help. The reported payoff: up to 3× faster throughput than comparable Transformers like Llama-3.1 and Qwen-2.5, while matching accuracy on MMLU, GSM8K, HumanEval, and MATH.
AI21's Jamba line put hybrids into production at scale, running hundreds of billions of parameters with a heavily Mamba-weighted attention-to-SSM ratio and long context windows — proof the architecture serves real traffic, not just papers.
The convergence. Multiple independent teams — NVIDIA, Alibaba's Qwen group, and the Mamba researchers themselves — landed on a strikingly similar recipe in 2026: roughly 75% linear (SSM) layers, 25% attention, plus a mixture-of-experts feed-forward. When separate labs chasing different goals arrive at the same blueprint, that's usually a sign the design space has a real optimum.
Why hybrids, not pure Mamba
If Mamba is so efficient, why keep any attention at all? Because the fixed-size state is both the feature and the limitation.
A constant-size state is a lossy compression of history. For most tokens, that's fine — you don't need perfect recall of everything to predict the next word. But some tasks demand exact retrieval from far back in the context: pulling a specific number from a long document, copying a variable name defined thousands of tokens earlier, precise in-context lookup. Pure attention nails these because it keeps everything. Pure SSMs can blur them.
The hybrid recipe is an engineering compromise made explicit: use cheap SSM layers for the 90%+ of work that only needs a running summary, and spend a few expensive attention layers where exact global recall actually pays off. You get most of the efficiency with almost none of the retrieval penalty.
What this means going forward
The lesson of Mamba isn't that the Transformer is dead. It's that "attention is all you need" was an overstatement — attention is some of what you need, and for long sequences, it's an expensive way to get the rest.
For practitioners, three implications stand out:
- Long-context economics are shifting. As hybrids proliferate, the cost curve for 128K+ context bends downward. Applications that were priced out — whole-repository reasoning, long-document analysis, persistent agents — get cheaper to run.
- Architecture is a design choice again. For years, "build a Transformer" was the only answer. Now the ratio of SSM to attention is a tunable knob, and it will be tuned per use case.
- Watch the state, not just the parameters. In a hybrid, memory behavior is governed by how much attention survives. That, more than raw parameter count, will predict how well a model handles precise long-range recall.
The Bottom Line
Mamba didn't dethrone the Transformer — it did something more useful. By proving that a selective state space model can match attention's quality at a fraction of the cost, it forced the field to treat attention as one tool among several. The frontier's answer in 2026 wasn't to pick a side but to blend them, and the emerging consensus of mostly-SSM, some-attention, plus MoE looks less like a fad and more like the next default. The Transformer era isn't over. It's just no longer the only game in town.
