realm (retrieval-augmented language model)
**REALM (Retrieval-Augmented Language Model)** is a pre-training framework that jointly trains a neural knowledge retriever and a language model encoder, where the retriever learns to fetch relevant text passages from a large corpus (e.g., Wikipedia) and the language model learns to use the retrieved evidence to make better predictions. Unlike post-hoc retrieval augmentation, REALM trains the retriever end-to-end with the language model using masked language modeling as the learning signal.
**Why REALM Matters in AI/ML:**
REALM demonstrates that **jointly training retrieval and language understanding** produces models that explicitly ground their predictions in retrieved evidence, achieving superior performance on knowledge-intensive tasks while providing interpretable, verifiable reasoning.
• **End-to-end retrieval training** — The retriever (a BERT-based bi-encoder) is trained jointly with the language model through backpropagation; the retrieval score p(z|x) is treated as a latent variable, and the model marginalizes over the top-k retrieved documents to compute the final prediction
• **MIPS indexing** — Maximum Inner Product Search (MIPS) over pre-computed document embeddings enables retrieval from millions of passages in milliseconds; the document index is asynchronously refreshed during training as the retriever improves
• **Knowledge-grounded prediction** — For masked token prediction, the model retrieves relevant passages and conditions its prediction on the retrieved evidence: p(y|x) = Σ_z p(y|x,z) · p(z|x), where z ranges over retrieved documents
• **Salient span masking** — REALM preferentially masks salient entities and dates rather than random tokens, focusing pre-training on knowledge-intensive predictions that benefit most from retrieval augmentation
• **Scalable knowledge** — Instead of memorizing world knowledge in model parameters (requiring ever-larger models), REALM stores knowledge in a retrievable text corpus that can be updated, expanded, and audited independently of the model
| Component | REALM Architecture | Notes |
|-----------|-------------------|-------|
| Retriever | BERT bi-encoder | Embeds query and documents separately |
| Knowledge Source | Wikipedia (13M passages) | Updated asynchronously during training |
| Retrieval | MIPS (top-k, k=5-20) | Sub-linear time via ANN index |
| Reader | BERT encoder | Conditions on query + retrieved passage |
| Pre-training Task | Masked LM with retrieval | Salient span masking |
| Marginalization | Over top-k documents | p(y|x) = Σ p(y|x,z)·p(z|x) |
| Index Refresh | Every ~500 training steps | Asynchronous re-embedding |
**REALM pioneered the paradigm of jointly training retrieval and language modeling, demonstrating that end-to-end learned retrieval produces models that explicitly ground predictions in evidence from a knowledge corpus, achieving state-of-the-art performance on knowledge-intensive NLP benchmarks while providing interpretable and updatable knowledge access.**