sentence permutation

**Sentence Permutation** is a **pre-training objective where the order of sentences in a document is randomly shuffled, and the model must reconstruct the original order** — used in models like BART and PEGASUS to teach the model about document-level structure, coherence, and flow logic. **Permutation Mechanism** - **Shuffling**: Break document into sentences $S_1, S_2, dots, S_n$. Randomly permute them to $S_{p1}, S_{p2}, dots, S_{pn}$. - **Reconstruction**: The model (typically seq2seq) treats the shuffled text as input and must generate the sentences in the correct original order. - **Difficulty**: Extremely challenging for long documents — requires understanding logical progression, anaphora, and narrative arc. - **BART**: Uses sentence permutation as one of detailed denoising objectives. **Why It Matters** - **Coherence**: Forces the model to understand *why* sentence A follows sentence B — logic and causality. - **Summarization**: Excellent pre-training for summarization — requires understanding global document structure. - **Long Context**: Encourages attention to long-range dependencies across the entire input. **Sentence Permutation** is **unscrambling the story** — a document-level objective that forces the model to learn structure and coherence by reordering shuffled sentences.