masked language modeling

**Masked Language Modeling (MLM)** is the **pre-training objective introduced by BERT where a percentage of input tokens are hidden (masked), and the model must predict them using bidirectional context** — typically masking 15% of tokens and minimizing the cross-entropy loss of the prediction. **The "Cloze" Task** - **Input**: "The quick [MASK] fox jumps over the [MASK] dog." - **Target**: "brown", "lazy". - **Refinement**: 80% [MASK], 10% random token, 10% original token (to prevent mismatch between pre-training and fine-tuning). - **Efficiency**: Only 15% of tokens provide a learning signal per pass (unlike CLM where 100% do). **Why It Matters** - **Revolution**: Started the Transformer revolution in NLP (BERT) — smashed records on benchmarks (GLUE, SQuAD). - **Representation**: Creates deep, context-aware vector representations of words. - **Pre-training Standard**: Remains the standard for encoder-only models (BERT, RoBERTa, DeBERTa). **MLM** is **fill-in-the-blanks** — the bidirectional pre-training task that teaches models deep understanding of language structure and relationships.