random span length

**Random Span Length** is a **masking parameter used in span-based pre-training objectives (like SpanBERT and T5)** — instead of masking spans of a fixed size, the length of each masked span is sampled from a probability distribution (typically geometric or uniform) to expose the model to missing information of varying granularity. **Distribution Details** - **Geometric Distribution**: Most common choice (e.g., SpanBERT uses $l sim Geo(0.2)$) — skews toward shorter spans but allows occasional long spans. - **Mean Length**: Typically targeted around 3 subword tokens — balancing single words and short phrases. - **Clamping**: Spans are often clamped to a maximum length (e.g., 10) to prevent masking practically the entire sequence. - **Diversity**: Ensures the model learns to handle both local (short span) and global (long span) context reconstruction. **Why It Matters** - **Robustness**: Evaluating on variable-length missing information makes the representation more robust. - **Realism**: Real-world noise or missing data isn't fixed-length — random lengths simulate diverse corruption. - **Generalization**: Prevents the model from overfitting to a specific "missing hole size" heuristic. **Random Span Length** is **variable-sized holes** — sampling mask lengths from a distribution to train models on diverse reconstruction challenges.