block-recurrent transformer
**Block-Recurrent Transformer** is the **hybrid architecture that partitions input sequences into fixed-size blocks, applies full transformer self-attention within each block, and passes a learned recurrent state between blocks to propagate long-range context** — combining the high-quality local attention of transformers with the unbounded-length capability of recurrent networks, enabling processing of arbitrarily long sequences with bounded O(block_size²) memory per step.
**What Is a Block-Recurrent Transformer?**
- **Definition**: A sequence model that divides input into non-overlapping blocks of B tokens, applies standard multi-head self-attention within each block, and transmits a fixed-size recurrent state vector from one block to the next — the recurrent state carries compressed information from all previous blocks.
- **Within-Block**: Full transformer attention — every token in the block attends to every other token in the same block. This provides the rich, parallel, high-quality representations that transformers excel at.
- **Between-Block**: Recurrent state update — a learned function (cross-attention to previous state, or gated RNN-style update) compresses the current block's output into a state vector passed to the next block.
- **Bounded Memory**: Memory usage is O(B²) per block plus O(d_state) for the recurrent state — independent of total sequence length, enabling arbitrarily long inputs.
**Why Block-Recurrent Transformer Matters**
- **Infinite Context Length**: Unlike standard transformers with fixed context windows, block-recurrent models process sequences of any length — the recurrent state theoretically carries information from the entire history.
- **Bounded Compute Per Step**: Each block requires O(B²) attention compute — regardless of how many blocks have been processed before. This makes both training and inference costs predictable and controllable.
- **Best of Both Worlds**: Full transformer attention within blocks captures rich local interactions; recurrence between blocks captures long-range dependencies — combining the strengths of both paradigm families.
- **Streaming Capability**: Can process input as a stream of blocks without storing the full sequence — suitable for real-time applications where input arrives continuously.
- **Memory-Efficient Training**: Gradient computation requires storing only O(number_of_blocks × d_state) recurrent states rather than the full O(sequence_length × d_model) activation cache.
**Block-Recurrent Architecture**
**Forward Pass Per Block**:
- Input: block of B tokens + recurrent state from previous block.
- Cross-attention: block tokens attend to previous recurrent state (context injection).
- Self-attention: standard multi-head attention within the B tokens.
- State update: compress block output into new recurrent state via attention pooling or gated combination.
- Output: processed B tokens + updated recurrent state.
**Recurrent State Mechanisms**:
- **Cross-Attention State**: Fixed number of state vectors; new block cross-attends to state for context, then state is updated via cross-attention from state to block output.
- **Gated State Update**: s_new = gate × s_old + (1 − gate) × compress(block_output) — similar to LSTM/GRU update.
- **Memory-Augmented**: State includes a small memory matrix that tokens can read from and write to — richer state representation.
**Comparison With Other Long-Context Methods**
| Method | Context | Compute/Step | Parallelizable | State |
|--------|---------|-------------|---------------|-------|
| **Full Transformer** | Fixed window | O(n²) | Fully parallel | None |
| **Transformer-XL** | Window + cache | O(n × (n+cache)) | Parallel within window | Cache |
| **Block-Recurrent** | Unbounded | O(B²) | Parallel within block | Recurrent state |
| **Pure RNN (Mamba)** | Unbounded | O(n) | Sequential | Recurrent state |
Block-Recurrent Transformer is **the architectural bridge between the transformer and recurrent paradigms** — partitioning the challenging problem of long-range sequence modeling into a solved local problem (transformer attention within blocks) and a manageable global problem (recurrent state between blocks), achieving unbounded context with bounded resources.