recurrent llm
**Recurrent LLM Architectures (RWKV, Mamba)** are **models that achieve linear-time sequence processing by replacing quadratic self-attention with recurrent or state-space mechanisms**, enabling efficient processing of very long sequences while maintaining competitive quality with transformer-based LLMs — reviving recurrent approaches at the billion-parameter scale.
**The Transformer Bottleneck**: Standard self-attention has O(N²) time and memory complexity in sequence length N. Even with Flash Attention (O(N) memory), the O(N²) compute remains. For sequence lengths of 100K-1M+ tokens, this quadratic cost becomes prohibitive. Recurrent architectures process sequences in O(N) time with O(1) memory per step.
**RWKV (Receptance Weighted Key Value)**:
| Component | Mechanism | Purpose |
|-----------|----------|--------|
| **Time-mixing** | WKV attention with linear complexity | Sequence mixing (replaces attention) |
| **Channel-mixing** | Gated FFN with shifted tokens | Feature interaction |
| **Token shift** | Linear interpolation with previous token | Local context injection |
RWKV replaces softmax attention with a weighted sum that can be computed recurrently: wkv_t = (Σ e^(w_s + k_s) · v_s) / (Σ e^(w_s + k_s)) where w provides exponential decay weights. This is computable as a running sum (RNN mode) or as a parallelizable scan (training mode). RWKV scales to 14B+ parameters with quality approaching transformer LLMs of similar size.
**Mamba (Selective State Space Model)**:
Mamba builds on structured state space models (S4) but adds **input-dependent (selective) parameters**: the state transition matrices A, B, C vary based on the input at each step, enabling the model to selectively remember or forget information — unlike time-invariant SSMs where the same dynamics apply regardless of input content.
**Mamba Architecture**: Each Mamba block contains: a selective SSM layer (replaces attention), a gated MLP path, and residual connections. The selective SSM: h_t = A_t · h_{t-1} + B_t · x_t, y_t = C_t · h_t, where A_t, B_t, C_t are functions of the input x_t. This selectivity is crucial — it allows the model to decide what to store in its fixed-size state based on input content.
**Training Efficiency**: Despite being recurrent at inference, both RWKV and Mamba use **parallel scan algorithms** during training: the recurrence h_t = A_t · h_{t-1} + B_t · x_t is a linear recurrence that can be parallelized using the associative scan primitive, computing all hidden states in O(N log N) time on GPUs. This provides transformer-like training parallelism with RNN-like inference efficiency.
**Inference Advantage**:
| Aspect | Transformer | Mamba/RWKV |
|--------|------------|------------|
| Generation per token | O(N) (KV cache lookup) | O(1) (fixed state update) |
| Memory per token | O(N) (growing KV cache) | O(d²) (fixed state size) |
| Prefill cost | O(N²) | O(N) |
| Long context cost | Grows linearly with N | Constant |
**Quality Comparison**: Mamba-2 (2024) matches transformer quality on language modeling up to ~3B parameters. At larger scales, pure recurrent models show a small but persistent gap on tasks requiring precise long-range retrieval (finding a specific fact buried deep in context). Hybrid architectures (interleaving attention and Mamba layers) close this gap while retaining most efficiency benefits.
**Recurrent LLM architectures represent a fundamental challenge to the transformer's dominance — demonstrating that linear-time sequence models can achieve competitive quality while offering dramatically better inference efficiency for long sequences, potentially enabling a new generation of models that process books, codebases, and video streams as native context.**