static masking

**Static Masking** is the **original masking strategy used in BERT, where masking patterns were generated once during data preprocessing and fixed** — meaning the model saw identical masked inputs for the same sentence across all training epochs. **The Process** - **Preprocessing**: Read corpus → Tokenize → Apply 15% masks → Save as TFRecords/HDF5. - **Training**: Load saved records. Epoch 1 and Epoch 10 see identical `[MASK]` positions. - **Limitation**: If training for 40 epochs, the model memorizes "Input A always has token 4 masked". It limits the variety of training signals. **Why It Matters** - **Historical Context**: Was an efficiency choice in original BERT (preprocessing is expensive). - **Obsolescence**: Replaced by Dynamic Masking (RoBERTa) which proved superior. - **Lesson**: Data augmentation (variation) usually beats static data in deep learning. **Static Masking** is **fixed-pattern training** — a legacy approach where the training questions (masks) never changed, limiting the model's learning potential.