span masking

**Span Masking** is a **pre-training strategy for masked language models where contiguous sequences (spans) of tokens are masked instead of individual random tokens** — popularized by SpanBERT and T5, this approach forces the model to predict entire phrases using only the surrounding context, encouraging the learning of longer-range dependencies and phrasal semantics. **Span Masking Details** - **Contiguous Spans**: Instead of masking single tokens, mask a sequence like "New York City" as [MASK] [MASK] [MASK] or a single [MASK] token. - **Geometric Distribution**: Span lengths are often sampled from a geometric distribution (e.g., mean length 3) — favoring short phrases but allowing longer ones. - **Objective**: Usually combined with a span boundary objective (SBO) or standard MLM stability. - **T5 Approach**: T5 replaces the entire span with a single sentinel token (e.g., ) and trains the model to generate the missing span. **Why It Matters** - **Harder Task**: Predicting a span is harder than predicting a single token — reduces the reliance on shallow local cues. - **Downstream Performance**: Significantly improves performance on span-selection tasks like Question Answering and Coreference Resolution. - **Efficiency**: Can be more sample-efficient than single-token masking for learning structural relationships. **Span Masking** is **hiding chunks of text** — forcing the model to reconstruct entire phrases from context, fostering deeper semantic understanding.