attention-ssm

Mamba-2 is the 2024 successor to the Mamba selective state-space model, and its importance is less about a bigger benchmark number than about a unifying idea: it shows that state-space models and attention are two views of the same underlying computation. That result, called state-space duality (SSD), lets a Mamba layer be computed with the same dense matrix multiplications that make attention fast on modern accelerators — reclaiming the tensor-core efficiency that the original Mamba's custom scan gave up. Alongside it, "hybrid attention-SSM" architectures like Jamba interleave a few attention layers among many SSM layers, keeping linear-time long-context scaling while buying back the one thing pure SSMs are bad at: exact recall.\n\n**Mamba-2's central claim is a duality — the selective SSM and attention are two sides of one structured-matrix computation.** Any state-space model can be written as multiplication by a large matrix that is *semiseparable*: its entries are determined by a low-rank recurrence, so the matrix never has to be formed in full. Attention, meanwhile, is already a matrix operation (softmax of QKᵀ). SSD makes the correspondence precise: a linear-attention-style computation with a particular structured mask *is* a state-space model, and vice versa. This means the same layer can be run two ways — as a linear-time recurrence for generation, or as a quadratic-but-parallel matmul for training — choosing whichever is cheaper for the hardware and the phase.\n\n**The practical prize of that duality is hardware efficiency: Mamba-2 runs on tensor cores, Mamba-1 largely did not.** The original Mamba used a hand-written associative scan that, while linear in sequence length, mapped poorly onto the matrix-multiply units that dominate GPU and TPU FLOPs. By restricting the state-transition to a scalar-times-identity form, Mamba-2 exposes the computation as block matrix multiplications (the SSD algorithm), letting it use the same accelerator paths as attention and reach roughly two-to-eight times the training throughput of Mamba-1 — while also allowing a much larger internal state dimension, which improves quality.\n\n**Pure SSMs have one structural weakness: they compress the entire past into a fixed-size state, so exact recall is hard.** A Transformer keeps every previous token in its KV cache and can attend back to any of them precisely, which is why it excels at copying, in-context retrieval, and induction. An SSM instead summarizes history in a constant-size hidden state, so its memory of any specific earlier token fades — cheap and constant-memory, but lossy for tasks that need to fetch an exact token from far back. This recall gap, not raw language modeling loss, is the main reason nobody has fully replaced attention with SSMs.\n\n**Hybrid attention-SSM models resolve the tension by interleaving a small number of attention layers among many SSM layers.** Jamba (a Transformer-Mamba mixture-of-experts model) uses roughly one attention layer for every seven Mamba layers, so the bulk of the network enjoys linear-time, constant-memory long-context processing while the sparse attention layers restore precise recall. Others follow the same recipe with different attention flavors — sliding-window attention interleaved with Mamba, or shared attention blocks — all trading a little of the SSM's efficiency for the retrieval ability that made attention indispensable in the first place. The emerging consensus is not "SSM versus Transformer" but a blend tuned to the context length and recall demands of the task.\n\n| Architecture | Per-token state | Sequence scaling | Exact recall | Accelerator fit |\n|---|---|---|---|---|\n| Transformer | Grows with context (KV cache) | Quadratic | Excellent | Tensor cores (attention matmuls) |\n| Mamba-1 | Constant (selective SSM) | Linear | Weak | Custom scan, under-uses tensor cores |\n| Mamba-2 | Constant, larger state | Linear (train as matmul) | Weak-to-fair | Tensor cores via SSD |\n| Hybrid (Jamba) | Mostly constant + sparse KV | Near-linear | Strong (attention layers) | Tensor cores throughout |\n\n```svg\n\n```\n\nThe wrong way to file Mamba-2 is as the next entry in a Transformer-versus-SSM horse race. The right way is to take its core result at face value: attention and state-space models are not rival architectures but two computations of the same structured operator, and once you see that, the design space opens up. You can run the operator as a linear recurrence when you want cheap generation, as a tensor-core matmul when you want fast training, and — because pure SSMs pay for their constant-size state with weak recall — you can splice in a handful of real attention layers exactly where precise retrieval matters, as Jamba and its kin do. Read Mamba-2 through a state-space-and-attention-are-one-computation lens rather than a which-architecture-wins lens, and the duality, the tensor-core speedup, and the attention-SSM hybrids stop looking like three separate results and become one: sequence mixing is a structured matrix, and you get to choose how to compute it and how much exact memory to pay for.