token deletion

**Token Deletion** is a **simple denoising objective where random tokens are deleted from the input sequence** — unlike masking (which typically replaces tokens with a [MASK] symbol), deletion removes the token entirely, changing the sequence length and forcing the model to infer missing positions without explicit markers. **Deletion Details** - **Process**: Iterate through sequence, delete token $t_i$ with probability $p$. - **No Placeholder**: The resulting sequence is shorter. The model doesn't know *where* tokens are missing. - **Difficulty**: Harder than masking because the *position* of the missing info is also unknown. - **BART**: Uses token deletion as one of its pre-training transformations. **Why It Matters** - **Robustness**: Makes the model robust to dropped words or transmission errors. - **Real-World Noise**: ASR (speech recognition) and typing errors often involve omissions, not just substitutions. - **Structure**: Forces the model to learn grammatical structure to realize "something is missing here." **Token Deletion** is **missing words without a trace** — a denoising task where the model must rewrite text to restore words that were completely removed.