recurrent memory transformer
**Recurrent Memory Transformer (RMT)** is a transformer architecture augmented with a set of dedicated memory tokens that are prepended to the input sequence and propagated across segments, enabling the model to maintain and update persistent memory across arbitrarily long sequences without modifying the core transformer attention mechanism. Memory tokens are read and written through standard self-attention, providing a natural interface between the working context and long-term stored information.
**Why Recurrent Memory Transformer Matters in AI/ML:**
RMT enables **effectively unlimited context length** by propagating compressed memory tokens across fixed-length segments, combining the efficiency of segment-level processing with the ability to retain information across millions of tokens.
• **Memory token mechanism** — A fixed set of M special tokens (typically 5-20) are prepended to each input segment; after processing through all transformer layers, the updated memory tokens carry forward to the next segment as compressed representations of all previously processed content
• **Segment-level processing** — The input sequence is divided into fixed-length segments (e.g., 512 tokens); each segment is processed with the memory tokens from the previous segment, enabling linear-time processing of arbitrarily long sequences
• **Read-write through attention** — Memory tokens participate in standard self-attention within each segment: "reading" occurs when input tokens attend to memory tokens, "writing" occurs when memory tokens attend to input tokens and update their representations
• **Backpropagation through memory** — Gradients can flow through the memory tokens across segments during training, enabling the model to learn what information to store, update, and retrieve from memory for downstream tasks
• **No architectural changes** — RMT works with any pre-trained transformer by simply adding memory tokens and fine-tuning, making it a practical approach to extending context length without retraining from scratch
| Feature | RMT | Standard Transformer | Transformer-XL |
|---------|-----|---------------------|----------------|
| Context Length | Unlimited (via memory) | Fixed (context window) | Extended (segment recurrence) |
| Memory Type | Learned tokens | None (attention only) | Cached hidden states |
| Memory Size | M tokens × d_model | N/A | Segment length × d_model |
| Compression | High (M << segment length) | None | None (full states cached) |
| Training | BPTT through memory | Standard | Truncated BPTT |
| Inference Memory | O(M × d) per segment | O(N² × d) | O(L × N × d) |
**Recurrent Memory Transformer provides a practical, architecture-agnostic approach to extending transformer context length to millions of tokens by propagating a compact set of learned memory tokens across input segments, enabling efficient long-range information retention and retrieval through standard self-attention without any modifications to the core transformer architecture.**