span boundary objective

**Span Boundary Objective (SBO)** is a **pre-training objective introduced in SpanBERT where the model must predict a masked span using ONLY the tokens at the span's boundaries** — ensuring that the representation of boundary tokens captures the semantics of the entire contents between them. **Mechanism** - **Task**: For a masked span $(x_s, dots, x_e)$, predict each token $x_i$ in the span. - **Input**: The tokens at the boundaries $x_{s-1}$ and $x_{e+1}$ PLUS the position embedding of the target $x_i$. - **Formula**: $P(x_i) = f(x_{s-1}, x_{e+1}, pos_i)$. - **Constraint**: The model CANNOT use the other masked tokens inside the span for context — only the edges. **Why It Matters** - **Span Representations**: Forces the boundary tokens to summarize the span content. - **Extractive Tasks**: Extractive QA and Span Selection rely on start/end pointers — SBO directly optimizes these boundary representations. - **Performance**: SpanBERT with SBO set SOTA on SQuAD and other span-based benchmarks. **Span Boundary Objective** is **judging a book by its covers** — forcing the model to reconstruct a phrase using only the words immediately before and after it.