mask-predict

**Mask-Predict** is a **non-autoregressive text generation strategy that iteratively predicts masked tokens** — starting from a fully masked sequence, the model predicts all tokens simultaneously, then masks the least confident predictions and re-predicts them, repeating for a fixed number of iterations. **Mask-Predict Algorithm** - **Initialize**: Start with a fully masked sequence of predicted length N: [MASK] [MASK] ... [MASK]. - **Predict**: Generate all tokens simultaneously using a conditional masked language model. - **Mask**: Mask the $k$ tokens with the lowest prediction confidence — $k$ decreases each iteration. - **Repeat**: Re-predict the masked positions conditioned on the unmasked tokens — iterate T times (typically 4-10). **Why It Matters** - **CMLM**: Introduced by Ghazvininejad et al. (2019) for machine translation — dramatically faster than autoregressive decoding. - **Quality**: 4-10 iterations achieve quality competitive with autoregressive translation — far fewer computation steps. - **Confidence-Based**: Masking low-confidence tokens focuses computation where it's most needed — efficient refinement. **Mask-Predict** is **confident tokens stay, uncertain ones retry** — iteratively improving generated text by re-predicting the least confident token positions.