Mamba: What If Attention Isn't the Answer?

There are two ways to follow a long meeting.

You can write down every single word. Perfect recall — ask about anything and you can look it up exactly. But re-reading gets slower and slower, and by hour four your transcript is enormous.

Or you can take running notes. Constant effort per minute, and the notes never grow past one page. But whatever you did not write down is gone forever.

Today's language models take the first approach. Attention lets the model relate every word to every other word, which is a perfect transcript — and it costs. Double the length and the work goes up four times. Worse, while the model writes an answer it keeps a growing memory of everything so far, so the thousandth word costs more than the first.

The older approach — running notes — is what recurrent networks did. Steady cost per word, fixed-size memory. They lost anyway, for two reasons: they could not be trained fast on modern hardware, and they forgot things.

Mamba is an attempt to get the running-notes cost with something much closer to the transcript's memory.

See the research paper

Albert Gu and Tri Dao published Mamba: Linear-Time Sequence Modeling with Selective State Spaces in December 2023.

1. Running notes, written down properly

A state space model comes from control theory — the maths of describing how a system with an internal state responds to what happens to it. Written for a sequence, it is:

$$h_t = \bar{\mathbf{A}}h_{t-1} + \bar{\mathbf{B}}x_t, \qquad y_t = \mathbf{C}h_t$$

In words: your notes at this moment are your previous notes, adjusted a bit, plus something from the current word. Then the output is read off the notes.

That looks exactly like the old recurrent networks, with one crucial difference: there is no squashing step between one moment and the next. The update is a plain weighted sum.

That sounds like a technical footnote. It is the whole reason this approach became trainable at scale. Because the update is a plain weighted sum, the entire sequence can be worked out in one big parallel operation instead of one word at a time — so the model trains fast on a modern chip, and then runs word-by-word with fixed-size notes when you actually use it.

2. The problem: the same note-taking for every word

In the earlier versions of this idea, the adjustment applied at each step was fixed — learned once during training and then identical for every word in every sentence.

That fixed-ness is what made the parallel training trick possible. It is also a hard ceiling.

The model cannot decide that this word matters and should be written down firmly, while that one is filler and can be ignored. It compresses history the same way regardless of content. It is a note-taker following a rigid rule rather than paying attention.

The paper shows the gap with two deliberately simple tasks. Selective copying asks the model to reproduce certain items while ignoring junk mixed in among them — you have to choose what to remember. Induction heads asks it to spot a pattern from earlier and complete it — you have to look something up based on its content.

Attention does both easily, because it compares everything to everything. A fixed note-taker fails both, and no amount of scale fixes it, because the shortcoming is structural.

3. The fix: let the input decide

Mamba's change is direct. Make the adjustment depend on the word currently being read, instead of being fixed.

Now the note-taking is shaped by what is coming in. The model can learn to take a big step for an important word — writing it firmly into the notes — and a tiny step for an unimportant one, effectively skipping it. It can decide, word by word, what goes into the notes and what gets read back out.

This is a gating mechanism, and it is closely related to what made LSTMs work back in the 1990s. The difference is that it is embedded in a plain weighted-sum update built for modern hardware, rather than in a squashed one that cannot be parallelised.

Both of those synthetic tasks are solved once selectivity is added. That is the paper's central experiment: the failures were caused by fixed note-taking, and letting the input steer it removes them.

4. The cost: the fast training trick just broke

There is an immediate and severe problem.

If the adjustment changes at every step, the whole-sequence-at-once shortcut no longer applies. There is no single fixed operation to apply. The entire reason these models were trainable at scale has just been destroyed.

Fixing that is the paper's second contribution, and it is an engineering contribution rather than a mathematical one. Three ideas together.

A parallel scan. Even with changing adjustments, the update has a helpful property: you can combine chunks of it in any grouping and get the same answer. That means you can compute it in a tree rather than a line — the same trick that lets a computer add up a million numbers in twenty steps instead of a million. So the sequence can still be processed in parallel, just differently.

Keeping it on the worktop. The expanded notes are large, and shuttling them to main memory would make the model wait on data rather than arithmetic. So the whole scan is fused into a single operation that keeps the notes in the chip's small fast memory, reads the input once, and writes only the outputs.

Recomputing instead of remembering. The intermediate notes needed for training are not saved. They are recomputed when needed, trading extra arithmetic for avoiding memory trips.

Anyone who has read the FlashAttention paper will recognise all three, which is unsurprising given the shared author. The pattern is identical: measure the memory hierarchy, keep the big thing in fast memory, recompute rather than store.

5. What it achieves

From the abstract:

• 5× higher throughput than Transformers when generating
• Linear scaling with length — double the text, double the work, not quadruple — with performance still improving on sequences up to a million items long
• Mamba-3B beats Transformers of the same size and matches Transformers twice its size, both during training and on downstream tests
• Validated across language, audio and genomics

The architecture also simplifies: Mamba removes attention and the feed-forward blocks entirely, using a single repeated block type rather than the alternating structure of a Transformer.

That multi-modality result deserves emphasis. Audio and DNA involve sequences far longer than text, where the transcript approach is simply not an option. In those domains the steady cost is not a nice-to-have — it is the entire reason the model is usable at all.

6. What attention still does better

Mamba compresses history into fixed-size notes. That is exactly why it is efficient, and exactly what it costs.

Attention keeps everything and can look up any previous word precisely. Ask a Transformer about an arbitrary detail from 40,000 words ago, and the record of that word is still sitting there. Ask Mamba, and the answer depends on whether that detail survived being compressed into the notes — and if it did not, it is gone.

This shows up as weaker performance on tasks needing precise recall from long documents, which is a real and commercially important category: answering from long supplied documents, finding one specific fact in a haystack, code that refers back to a definition far above.

Selectivity narrows the gap by letting the model choose what to keep. It cannot close it, because fixed-size notes hold a fixed amount of information, and no amount of cleverness about what to write changes how much fits.

7. Where this actually landed

The most interesting outcome has not been Mamba replacing Transformers. It has been hybrids.

If attention is precise and expensive while notes are cheap and lossy, a model does not have to pick one globally. It can alternate — a few attention layers for precise lookup, note-taking layers for the bulk of the work. Several production models now do exactly this, and the results suggest the combination beats either pure approach at a given cost.

That is a more useful outcome than a clean replacement would have been, and it reframes the question the paper opened. Not is attention necessary but how much attention is necessary, and where — which is an engineering question with a tunable answer rather than a loyalty test.

8. The idea worth carrying

Mamba's real argument is that two properties everybody assumed were welded together can be separated.

Content-dependent processing — the thing attention gives you, where what the model does depends on what it is reading — does not have to cost quadratic time. It costs quadratic time in attention, because attention achieves it by comparing everything to everything. A note-taker whose notes vary with the input achieves the same selectivity in linear time, provided somebody is willing to write the low-level code that makes it fast.

Whether fixed-size notes prove to be a fatal ceiling or an acceptable trade is still open. What is settled is that the quadratic cost was a property of one particular mechanism, not a law about processing sequences — and demonstrating that clearly is worth a great deal, even if this specific architecture ends up as one component among several.

Sources

• Gu, Dao — Mamba: Linear-Time Sequence Modeling with Selective State Spaces, 2023
• Gu, Goel, Ré — Efficiently Modeling Long Sequences with Structured State Spaces (S4), 2021
• Dao, Gu — Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2), 2024