next sentence prediction

**Next Sentence Prediction (NSP)** is a **pre-training objective introduced in BERT where the model predicts whether a given sentence B immediately follows sentence A in the original text** — a binary classification task designed to teach the model relationships between sentences (discourse, entailment, continuity). **NSP Details** - **Input**: Pairs of sentences (A, B) packed together: `[CLS] A [SEP] B [SEP]`. - **Positive Sample (IsNext)**: B is the actual next sentence from the corpus (50% probability). - **Negative Sample (NotNext)**: B is a random sentence from the corpus (50% probability). - **Prediction**: The `[CLS]` token embedding is fed to a classifier to output IsNext/NotNext. - **Critique**: Later research (RoBERTa) showed NSP was not very effective — mostly learning topic matching rather than coherence. **Why It Matters** - **Original BERT**: A core component of the original BERT training recipe. - **Discourse**: Intended to help with tasks like QA and NLI (Natural Language Inference) that require reasoning across sentences. - **Legacy**: Largely replaced by more effective objectives (like SOP) or removed entirely in modern LLMs. **NSP** is **original BERT's coherence check** — a binary task checking if two sentences belong together, now considered largely obsolete by improved methods.