adaptive masking

**Adaptive Masking** refers to **strategies where the masking rate or pattern changes during training based on the model's performance or curriculum** — focusing learning on "hard" tokens or adjusting difficulty as the model improves. **Approaches** - **Hardness-based**: Mask tokens the model is currently predicting *well* (too easy) or *poorly* (need focus)? Typically, masking "hard" or "salient" tokens is better. - **Rate Scheduling**: Start with low masking rate (easy), increase to high masking rate (hard). - **Model-based**: Use a smaller model to identify "important" tokens to mask for a larger model. **Why It Matters** - **Efficiency**: Don't waste compute predicting "the", "a", "is" (easy stop words). - **Learning**: Force the model to solve difficult semantic relations. - **Complexity**: Adds complexity to the training pipeline — simple random masking is often "good enough" and surprisingly hard to beat. **Adaptive Masking** is **smart masking** — changing *what* or *how much* to hide based on what the model already knows.