summary generation as pre-training

**Summary Generation as Pre-training** (or Gap Sentence Generation) is a **pre-training strategy where the model learns to generate a summary of the input text** — either using naturally occurring summaries (headlines, abstracts) or pseudo-summaries created by identifying key sentences in the document (PEGASUS). **Data Sources** - **PEGASUS (GSG)**: Mask important sentences (those with high ROUGE overlap with the rest) and generate them. - **News Headlines**: Predict the headline from the article body. - **Abstracts**: Predict the abstract from the paper body. - **Reddit**: Predict the post title or TL;DR from the body. **Why It Matters** - **Abstraction**: Forces the model to synthesize information, not just copy it. - **Importance Ranking**: To summarize, the model must decide what is *important*. - **Downstream Alignment**: This objective aligns pre-training directly with the downstream task of abstractive summarization. **Summary Generation as Pre-training** is **learning to condense** — teaching the model to extract and synthesize the core meaning of a document.