text infilling

**Text Infilling** is a **pre-training objective where the model learns to generate missing spans of text at arbitrary positions** — used in models like BART and T5, it generalizes standard language modeling (predict next token) and masked language modeling (predict missing token) to the generation of variable-length missing sequences. **Infilling vs. MLM** - **MLM (BERT)**: Predicts a single token for each [MASK]. Structure is preserved. - **Infilling (T5/BART)**: Replaces a span of *any* length with a single unique sentinel/mask token. The model must predict the *entire* original span. - **Generation**: Requires a decoder or a seq2seq architecture — the output length is unknown and must be generated. - **Flexibility**: Can reconstruct a single word, a phrase, or a whole sentence from a single mask. **Why It Matters** - **Generative Capability**: Teaches the model to *generate* fluent text, not just classify tokens — essential for summarization and translation. - **Compression**: T5 uses infilling to frame all NLP tasks as "text-to-text" — extremely versatile. - **Code Generation**: Highly effective for code completion (infilling code blocks). **Text Infilling** is **filling in the blanks with generation** — predicting complete missing text spans rather than just classifying missing tokens.