REALM (Retrieval-Augmented Language Model) is a pre-training framework that jointly trains a neural knowledge retriever and a language model encoder, where the retriever learns to fetch relevant text passages from a large corpus (e.g., Wikipedia) and the language model learns to use the retrieved evidence to make better predictions. Unlike post-hoc retrieval augmentation, REALM trains the retriever end-to-end with the language model using masked language modeling as the learning signal.

Why REALM Matters in AI/ML: REALM demonstrates that jointly training retrieval and language understanding produces models that explicitly ground their predictions in retrieved evidence, achieving superior performance on knowledge-intensive tasks while providing interpretable, verifiable reasoning.

• End-to-end retrieval training — The retriever (a BERT-based bi-encoder) is trained jointly with the language model through backpropagation; the retrieval score p(z|x) is treated as a latent variable, and the model marginalizes over the top-k retrieved documents to compute the final prediction • MIPS indexing — Maximum Inner Product Search (MIPS) over pre-computed document embeddings enables retrieval from millions of passages in milliseconds; the document index is asynchronously refreshed during training as the retriever improves • Knowledge-grounded prediction — For masked token prediction, the model retrieves relevant passages and conditions its prediction on the retrieved evidence: p(y|x) = Σ_z p(y|x,z) · p(z|x), where z ranges over retrieved documents • Salient span masking — REALM preferentially masks salient entities and dates rather than random tokens, focusing pre-training on knowledge-intensive predictions that benefit most from retrieval augmentation • Scalable knowledge — Instead of memorizing world knowledge in model parameters (requiring ever-larger models), REALM stores knowledge in a retrievable text corpus that can be updated, expanded, and audited independently of the model

REALM pioneered the paradigm of jointly training retrieval and language modeling, demonstrating that end-to-end learned retrieval produces models that explicitly ground predictions in evidence from a knowledge corpus, achieving state-of-the-art performance on knowledge-intensive NLP benchmarks while providing interpretable and updatable knowledge access.