entity masking

**Entity Masking** is a **masking strategy that preferentially masks named entities (people, organizations, locations, dates) during pre-training** — targeting semantically important spans rather than random tokens, forcing the model to learn world knowledge and entity-level understanding. **Entity Masking Approach** - **Entity Detection**: Use NER (Named Entity Recognition) to identify entities in the training text. - **Preferential Masking**: Mask entire entities more frequently than random tokens — focus learning on factual knowledge. - **Entity Types**: Person names, organization names, locations, dates, quantities — semantically meaningful spans. - **ERNIE**: Baidu's ERNIE (Enhanced Representation through Knowledge Integration) popularized entity and phrase masking. **Why It Matters** - **Knowledge Acquisition**: Entity masking forces the model to memorize and reason about real-world entities — better knowledge representation. - **Downstream Tasks**: Improves performance on knowledge-intensive tasks — question answering, relation extraction, entity typing. - **Knowledge Graphs**: Can be combined with knowledge graph embeddings for enhanced entity understanding. **Entity Masking** is **hiding the important names** — forcing the language model to learn world knowledge by preferentially masking named entities during pre-training.