document rotation

**Document Rotation** is a **pre-training objective where a token is chosen uniformly at random, and the document is rotated so that token becomes the start** — used in models like BART, this trains the model to identify the true start of a document and reconstruct the original sequence. **Mechanism** - **Select Pivot**: Choose a random token $t_k$ in the sequence. - **Rotate**: Move tokens before $t_k$ to the end: $[t_k, dots, t_n, t_1, dots, t_{k-1}]$. - **Objective**: The model (seq2seq) must accept the rotated sequence and generate the original un-rotated sequence. - **Inference**: The model learns to be invariant to the starting point or to identify the logical beginning. **Why It Matters** - **Start Identification**: Forces identification of the introductory sentence or logical opening. - **Context Cycle**: Ensures the model can handle context that wraps around — useful for sliding window approaches? - **Global Structure**: Like shuffling, it forces an understanding of the document's overall structure. **Document Rotation** is **finding the beginning** — a structural pre-training task where the model learns to identify the start of a document from a rotated version.