salient span masking

**Salient Span Masking** is a **domain-specific masking strategy (used in REALM, RetriBERT) where spans that are "salient" (named entities, dates, key technical terms) are masked preferentially** — specifically designed to force the model to look up external knowledge or learn facts, rather than just guessing common words. **Mechanism** - **Identification**: Use a tagger (NER) or frequency analysis (TF-IDF) to find "salient" terms. - **Masking**: Mask these terms. - **Purpose**: "The capital of France is [MASK]." -> Model MUST know/retrieve "Paris". Random masking "The [MASK] of France is Paris" is trivial grammar. **Why It Matters** - **RAG (Retrieval-Augmented Generation)**: Crucial for training retrievers — the retriever must find a document containing "Paris" to solve the mask. - **Question Answering**: Improves performance on Open-Domain QA. - **Fact Learning**: Shifts focus from syntax ("The cat sat on [MASK]") to semantics/facts. **Salient Span Masking** is **fact-checking tests** — specifically hiding the answers to factual questions to force the model to learn or retrieve knowledge.