sentence scrambling

**Sentence Scrambling** is a **pre-training objective where the sentences of a document are randomly reordered, and the model must identify the correct order or predict the position of a specific sentence** — similar to Sentence Permutation but often formulated as a classification or ranking task rather than generation. **Variants** - **Reordering**: Generatively reconstruct the document (BART). - **Binary Classification**: "Do these two sentences appear in this order?" (ALBERT SOP). - **Ranking**: "Which of these 5 candidates is the correct next sentence?" - **Position Prediction**: "What is the absolute position of this sentence in the document?" **Why It Matters** - **Structure Learning**: Forces learning of narrative structure (Introduction → Body → Conclusion). - **Long-Range Dependencies**: To order sentences correctly, the model must track entities and themes across the whole document. - **Coherence**: Essential for tasks like summarization and story generation where flow matters. **Sentence Scrambling** is **putting the story back together** — teaching the model document-level coherence by forcing it to reassemble jumbled sentences.