dynamic masking

**Dynamic Masking** is a **training strategy for Masked Language Models (like RoBERTa)** where the **mask pattern is generated on-the-fly every time a sequence is fed to the model**, rather than being generated once and saved (Static Masking) — allowing the model to see different versions of the same sentence with different masks over training epochs. **Dynamic vs. Static** - **Static (Original BERT)**: Data was masked once during preprocessing. The model saw the exact same mask pattern for "Sentence A" in Epoch 1, 2, 10. - **Dynamic (RoBERTa)**: Mask is applied in the data loader. Epoch 1: "The [MASK] brown...", Epoch 2: "The quick [MASK]...". - **Benefit**: Effectively multiplies the dataset size — the model never "memorizes" the specific mask solution. **Why It Matters** - **Performance**: RoBERTa showed that dynamic masking improves performance significantly over static masking. - **Epochs**: Allows training for more epochs without overfitting to specific masks. - **Standard Practice**: Now standard in almost all MLM training pipelines. **Dynamic Masking** is **reshuffling the problem** — changing which words are hidden every time the model studies a sentence to prevent memorization.