gradient-based masking

**Gradient-Based Masking** is a **technique that selects tokens to mask based on their influence on the loss gradient** — identifying tokens that are most critical for the model's current state or that provide the strongest training signal. **Mechanism** - **Saliency**: Compute gradients with respect to input tokens. High gradient = this token matters a lot. - **Selection**: Mask tokens with high gradients (force the model to find alternative paths to meaning) OR mask tokens that maximize expected loss. - **One-Shot**: Requires a backward pass to find masks, then another pass to train — computationally expensive (2x cost). **Why It Matters** - **Adversarial**: Acts like adversarial training — attacking the model's reliance on specific keywords. - **Interpretability**: Reveals which tokens the model relies on. - **Cost**: Usually too expensive for large-scale pre-training compared to random dynamic masking. **Gradient-Based Masking** is **mathematically targeted hiding** — using the model's own internal gradients to decide which words are most important to hide.