denoising objective

**Denoising Objective** is a **general class of self-supervised learning objectives where the model is trained to reconstruct a clean input from a corrupted (noisy) version** — fundamental to BERT (MLM), BART, T5, and Denoising Autoencoders, teaching the model the data distribution by learning to remove noise. **Common Corruptions (Noise)** - **Masking**: Hiding tokens ([MASK]). - **Deletion**: Removing tokens. - **Infilling**: Replacing spans with a single mask. - **Permutation**: Shuffling order. - **Rotation**: Rolling the sequence. - **Replacement**: Swapping tokens with random ones. **The Goal** - **Loss**: Minimize reconstruction error (Cross-Entropy) between generated/predicted output and original clean input. - **Manifold Learning**: By mapping noisy points back to data points, the model learns the "manifold" of structured language. - **Context Dependence**: To fix noise, the model must understand the context — syntax, semantics, and facts. **Denoising Objective** is **learning by fixing** — the core principle of modern NLP pre-training: corrupt the data and teach the model to repair it.