phrase masking

**Phrase Masking** is a **masking strategy that masks complete phrases or multi-word expressions during pre-training** — instead of masking individual tokens, entire meaningful phrases (noun phrases, verb phrases, prepositional phrases) are masked together, encouraging the model to learn phrase-level semantics. **Phrase Masking Approach** - **Phrase Detection**: Use constituency parsing, chunking, or n-gram frequency to identify meaningful phrases. - **Span Masking**: Mask the entire phrase as a single unit — all tokens in the phrase are masked simultaneously. - **SpanBERT**: Masks random contiguous spans (1-10 tokens) — a simplified form of phrase masking. - **ERNIE**: Combines entity masking and phrase masking — multiple granularities of masking. **Why It Matters** - **Compositional Semantics**: Phrase masking forces the model to understand how words compose into meaningful units. - **Longer Dependencies**: Predicting entire phrases requires understanding longer-range context than single tokens. - **Better Spans**: Models trained with phrase masking perform better on span-level tasks — extractive QA, NER, coreference resolution. **Phrase Masking** is **hiding whole phrases** — masking complete multi-word expressions to teach the model compositional phrase-level understanding.