sentence order prediction

**Sentence Order Prediction (SOP)** is a **pre-training objective introduced in ALBERT to replace Next Sentence Prediction (NSP)** — instead of predicting if two sentences are random or consecutive (topic matching), SOP makes the model predict which of two *consecutive* sentences came first, forcing it to learn coherence rather than just topic similarity. **SOP vs. NSP** - **NSP Weakness**: Negative examples are random sentences — easy to distinguish by topic (e.g., "Hockey" vs. "Cooking"). - **SOP Hardness**: Negative examples are the SAME two consecutive sentences but SWAPPED order (B then A). - **Task**: Given two segments, predict if order is (A, B) or (B, A). - **Focus**: Modeling coherence, logical flow, and discourse markers — mere topic matching is insufficient. **Why It Matters** - **Better Representations**: ALBERT showed SOP leads to significantly better performance on multi-sentence reasoning tasks (SQuAD, RACE, MNLI). - **Efficiency**: Harder task signal allows more efficient learning of inter-sentence relationships. - **Structure**: Forces the model to understand narrative flow and causal links. **SOP** is **fixing NSP** — a harder ordering task that forces the model to learn logical coherence instead of just topic matching.