text shuffling

**Text Shuffling** describes **a range of pre-training objectives involving the randomization of token or span order** — forcing the model to rely on semantic coherence rather than just local syntax to reconstruct the original text. **Shuffling Levels** - **Token Shuffling**: Randomly shuffle tokens within a small window (e.g., 3-5 tokens) — de-correlates local position. - **Span Shuffling**: Shuffle the order of spans or phrases. - **Sentence Shuffling**: Permute full sentences (Sentence Permutation). - **N-gram Shuffling**: Shuffle blocks of N-grams. **Why It Matters** - **De-noising**: Used in Denoising Autoencoders (DAE) and BART. - **Dependency Learning**: If "President" and "Obama" are shuffled, the model must know they go together regardless of order. - **Regularization**: Prevents the model from over-relying on strict sequential order (though in NLP order usually matters). **Text Shuffling** is **scrambling the message** — forcing the model to reassemble order from chaos based on semantic relationships.