State space models (SSMs), and the Mamba architecture in particular, are a family of sequence models that challenge the Transformer's dominance by processing sequences in linear time instead of quadratic. Where attention compares every token to every other token, an SSM carries a compact hidden state forward through the sequence like a recurrent network — but structured so that it can also be trained in parallel. The payoff is cheap scaling to very long sequences and constant memory per token at generation time, which is exactly where Transformers hurt most.\n\n``svg\n\n``\n\nThe core idea is a structured linear recurrence. An SSM maps an input sequence to an output through a hidden state that evolves one step at a time: the next state is a linear function of the previous state plus the new input, and the output is a linear readout of the state. This is the classical state-space formulation from control theory, adapted for deep learning. Because the update is linear and time-invariant, the same simple dynamics — described by a few learned matrices — summarize an arbitrarily long history in a fixed-size state.\n\nIts trick is having two equivalent forms. During training the time-invariant recurrence can be unrolled into a single global convolution over the whole sequence, which runs in parallel on a GPU just as efficiently as attention. During inference it runs in its recurrent form, updating one fixed-size state per token — so generation costs constant time and constant memory per step, with no ever-growing KV cache. Getting both the parallel-training and cheap-inference form from one model is what makes SSMs attractive.\n\nS4 solved long-range memory. The Structured State Space (S4) model introduced a special initialization of the state matrix (based on HiPPO theory) that lets the state retain information across tens of thousands of steps, letting it beat Transformers on long-range benchmarks. But S4 is time-invariant: it applies the same dynamics to every input regardless of content, so it cannot selectively focus on or ignore particular tokens the way attention can — a real weakness on language.\n\nMamba adds selectivity. Mamba makes the key parameters — the input, output, and step-size terms — functions of the current input, so the model can decide what to remember and what to forget based on content. This closes much of the gap with attention on language modeling. The catch is that input-dependent dynamics break the convolution shortcut, so Mamba uses a hardware-aware parallel "selective scan" that keeps the state in fast GPU memory. The result is linear scaling in sequence length with several-times-higher inference throughput than a comparable Transformer.\n\nIt is a strong complement, not yet a wholesale replacement. Linear cost and constant generation memory make SSMs compelling for very long sequences — genomics, audio, high-resolution signals, long-context language — but pure attention still leads at the frontier, and precise recall or copying from far back in the context remains a relative weak spot. In practice the popular pattern is hybrids that interleave a few attention layers with many Mamba layers, capturing most of the efficiency while keeping attention's exactness where it matters.\n\n| Aspect | Transformer (attention) | State space model (Mamba) |\n|---|---|---|\n| Cost in sequence length | O(n²) | O(n) |\n| Memory per generated token | grows with context (KV cache) | constant (fixed state) |\n| How tokens mix | all-pairs attention | a recurrence through one state |\n| Content-based selection | native to attention | Mamba: input-dependent Δ, B, C |\n| Relative weak spot | quadratic cost and memory | exact long-range recall / copying |\n\nRead Mamba through a selective-linear-recurrence lens rather than a cheaper-attention lens: the advance is not merely dropping the quadratic cost, but making a constant-size state's dynamics depend on the input, so the model can choose what to keep and what to discard while still training in parallel and generating in constant memory.\n