Text Infilling is a pre-training objective where the model learns to generate missing spans of text at arbitrary positions — used in models like BART and T5, it generalizes standard language modeling (predict next token) and masked language modeling (predict missing token) to the generation of variable-length missing sequences.

Infilling vs. MLM

• MLM (BERT): Predicts a single token for each [MASK]. Structure is preserved.
• Infilling (T5/BART): Replaces a span of any length with a single unique sentinel/mask token. The model must predict the entire original span.
• Generation: Requires a decoder or a seq2seq architecture — the output length is unknown and must be generated.
• Flexibility: Can reconstruct a single word, a phrase, or a whole sentence from a single mask.

Why It Matters

• Generative Capability: Teaches the model to generate fluent text, not just classify tokens — essential for summarization and translation.
• Compression: T5 uses infilling to frame all NLP tasks as "text-to-text" — extremely versatile.
• Code Generation: Highly effective for code completion (infilling code blocks).

Text Infilling is filling in the blanks with generation — predicting complete missing text spans rather than just classifying missing tokens.