whole word masking

**Whole Word Masking** is a **masking strategy for masked language model pre-training where entire words are masked rather than individual subword tokens** — when a word is selected for masking, ALL its constituent subword pieces (WordPiece, BPE) are masked together, preventing the model from trivially predicting a masked piece from its sibling pieces. **Whole Word Masking Details** - **Standard Masking**: BERT randomly masks individual tokens — a word like "playing" tokenized as "play" "##ing" might only mask "##ing." - **Problem**: If only "##ing" is masked, the model can trivially predict it from "play" — learns subword patterns, not semantics. - **WWM**: If "playing" is selected, BOTH "play" AND "##ing" are masked — forces the model to use context. - **Rate**: Still masks ~15% of the original words — but the token masking rate may differ. **Why It Matters** - **Better Representations**: WWM produces better contextualized representations — the model must learn deeper semantic understanding. - **BERT-WWM**: Google released Whole Word Masking variants of BERT — improved performance on downstream tasks. - **Chinese**: Particularly important for Chinese BERT — Chinese word segmentation makes token-level masking problematic. **Whole Word Masking** is **masking complete words, not fragments** — forcing the model to learn from context rather than subword co-occurrence patterns.