Memorizing Transformer is a transformer architecture augmented with an external key-value memory that stores exact token representations from past context, enabling the model to attend over hundreds of thousands of tokens by combining a standard local attention window with approximate k-nearest-neighbor (kNN) retrieval from a large non-differentiable memory. The approach separates what the model memorizes (stored verbatim in external memory) from how it reasons (learned attention over retrieved memories).

Why Memorizing Transformer Matters in AI/ML: Memorizing Transformer enables massive context extension (up to 262K tokens) by offloading long-term storage to an external memory while preserving the model's ability to precisely recall and attend over previously seen tokens.

• External kNN memory — Key-value pairs from past tokens are stored in a FAISS-like approximate nearest neighbor index; at each attention layer, the current query retrieves the top-k most relevant past tokens from memory, extending effective context to hundreds of thousands of tokens • Hybrid attention — Each attention head combines local attention (over the standard context window) with non-local attention (over kNN-retrieved memories), using a learned gating mechanism to weight the contribution of local versus retrieved information • Non-differentiable memory — The external memory is not updated through gradients; instead, key-value pairs are simply stored as the model processes tokens and retrieved as-is, eliminating the memory bottleneck of approaches that backpropagate through the full context • Exact recall — Unlike compressed or summarized memory representations, memorizing transformers store verbatim token representations, enabling exact retrieval of specific facts, rare entities, and long-range co-references • Scalable context — Memory size scales linearly with context length (just storing KV pairs), and kNN retrieval adds only O(k · log(N)) overhead per query, making 100K+ token contexts practical with standard hardware

Memorizing Transformer demonstrates that combining learned transformer attention with external approximate nearest-neighbor memory enables practical and effective context extension to hundreds of thousands of tokens, providing exact recall of distant information while maintaining computational efficiency through the separation of storage and reasoning mechanisms.