gap sentence generation

**Gap Sentence Generation (GSG)** is a **pre-training objective used in PEGASUS specifically designed for abstractive summarization** — whole sentences are masked (removed) from a document, and the model (seq2seq) must generate these missing sentences. **Mechanism** - **Selection**: Select "important" sentences (e.g., using ROUGE scores vs. the rest of the doc) to act as pseudo-summaries. - **Masking**: Remove these sentences from the input using [MASK1]. - **Generation**: The decoder must generate the exact text of the missing sentences. **Why It Matters** - **Summarization Bias**: Standard MLM doesn't teach summarization. GSG forces the model to synthesize content from the rest of the document. - **PEGASUS**: Showed that this objective beats standard BERT/Roberta approaches on summarization (CNN/DailyMail, XSum) with far less data. - **Principle**: Pre-training objectives should mimic the downstream task. **Gap Sentence Generation** is **summarization pre-training** — forcing the model to generate key missing sentences, simulating the abstractive summarization process.