mask token

**MASK token** is the **special token used to hide selected positions in text so models can learn contextual reconstruction objectives** - it is central to masked-language-model pretraining. **What Is MASK token?** - **Definition**: Reserved vocabulary symbol that replaces chosen tokens during training inputs. - **Training Objective**: Model predicts original hidden tokens from surrounding context. - **Model Family**: Most associated with encoder architectures such as BERT variants. - **Inference Difference**: Commonly used in pretraining tasks, not standard autoregressive decoding. **Why MASK token Matters** - **Context Learning**: Forces representations to capture bidirectional semantic dependencies. - **Sample Efficiency**: Generates supervised learning signal from unlabeled raw text. - **Transfer Performance**: Improves downstream quality on classification and extraction tasks. - **Protocol Consistency**: Correct mask-token ID mapping is required for reproducible training. - **Debug Value**: Mask prediction behavior helps inspect linguistic knowledge learned by models. **How It Is Used in Practice** - **Masking Policy**: Set masking ratio and replacement strategy for stable objective balance. - **Tokenizer Alignment**: Verify MASK token is defined and consistent across all training stages. - **Evaluation**: Track masked-token prediction accuracy and downstream transfer metrics. MASK token is **a core supervision primitive in encoder pretraining** - proper mask-token configuration directly influences representation quality.