retro (retrieval-enhanced transformer)

**RETRO (Retrieval-Enhanced Transformer)** is the **language model architecture that deeply integrates retrieval augmentation into the transformer by splitting input into chunks, retrieving relevant passages from a trillion-token database for each chunk, and conditioning generation on both the input and retrieved content through dedicated cross-attention layers** — demonstrating that a 7B parameter model with retrieval can match the performance of 25× larger dense models on knowledge-intensive tasks by offloading factual knowledge to an external database. **What Is RETRO?** - **Definition**: A transformer architecture with integrated retrieval — the input is split into fixed-size chunks (typically 64 tokens), each chunk triggers a nearest-neighbor search against a pre-built retrieval database, and retrieved passages are incorporated into generation via specialized chunked cross-attention (CCA) layers interleaved with standard self-attention. - **Chunked Cross-Attention (CCA)**: A novel attention mechanism where tokens in a chunk attend to the retrieved neighbors for that chunk — retrieved information is injected at specific points in the model rather than simply prepended to the context. - **Retrieval Database**: A pre-computed index of trillions of tokens (e.g., MassiveText corpus) encoded into dense embeddings by a frozen BERT encoder — enabling fast approximate nearest-neighbor retrieval at each chunk. - **Architecture Integration**: Retrieval is not a preprocessing step — it is woven into the model's forward pass, with CCA layers at every few transformer blocks enabling deep interaction between retrieved and generated content. **Why RETRO Matters** - **25× Parameter Efficiency**: RETRO-7B matches the perplexity of GPT-3 175B on knowledge-heavy tasks — demonstrating that retrieval substitutes for parametric memorization of facts. - **Updatable Knowledge**: The retrieval database can be updated without retraining the model — new facts, corrected information, and temporal knowledge can be inserted by updating the index. - **Reduced Hallucination**: By conditioning on retrieved factual content, RETRO generates text grounded in actual documents rather than relying solely on compressed parametric knowledge. - **Cost-Effective Scaling**: Scaling the retrieval database (adding more documents) is far cheaper than scaling model parameters — database storage costs pennies per GB while training compute costs millions per parameter doubling. - **Attribution**: Retrieved passages provide implicit citations for generated content — enabling source tracking that pure parametric models cannot provide. **RETRO Architecture** **Retrieval Pipeline**: - Split input into 64-token chunks: [c₁, c₂, ..., cₘ]. - For each chunk cᵢ, encode using frozen BERT → query embedding. - Retrieve top-k nearest neighbors from the pre-built FAISS index. - Each neighbor provides ~128 tokens of context surrounding the matched passage. **Chunked Cross-Attention (CCA)**: - Every third transformer block contains a CCA layer after the self-attention layer. - Tokens in chunk cᵢ cross-attend to the retrieved neighbors for cᵢ. - Retrieved content does not attend to the input (asymmetric attention). - CCA enables each generation chunk to be informed by relevant retrieved knowledge. **Training**: - Train with retrieval active — the model learns to use retrieved context from the start. - Frozen retriever (BERT) — only the main model and CCA weights are updated. - Loss is standard language modeling loss — retrieval improves predictions by providing relevant context. **RETRO Performance** | Model | Parameters | Retrieval | Perplexity (Pile) | Knowledge QA | |-------|-----------|-----------|-------------------|-------------| | **GPT-3** | 175B | None | Baseline | Baseline | | **RETRO** | 7.5B | 2T tokens DB | ≈ GPT-3 175B | ≈ GPT-3 | | **RETRO** | 7.5B | No retrieval | Much worse | Much worse | RETRO is **the architectural proof that knowledge storage and knowledge reasoning can be decoupled** — demonstrating that relatively small language models become powerful knowledge engines when coupled with massive retrieval databases, establishing the blueprint for the retrieval-augmented generation paradigm that now pervades production LLM systems.