sundae

**SunDAE** (Step-unrolled Denoising Autoencoder) is a **non-autoregressive text generation model that iteratively denoises a corrupted sequence** — starting from a randomly corrupted input and applying a denoising model repeatedly, with the key innovation of unrolling denoising steps during training for improved multi-step generation. **SunDAE Approach** - **Corruption**: Randomly corrupt the target sequence — random token replacement, masking, or insertion. - **Denoising**: Train a model to reconstruct the clean sequence from the corrupted version. - **Unrolled Training**: During training, perform multiple denoising steps and backpropagate through all steps — trains for iterative refinement. - **Generation**: At inference, start from random tokens and iteratively denoise — each step improves the output. **Why It Matters** - **Training-Inference Alignment**: Unrolled training aligns the training objective with the iterative inference procedure — reduces the train-test gap. - **Simple**: No complex scheduling or masking strategy needed — just corrupt, denoise, and repeat. - **Competitive**: Achieves competitive performance with more complex non-autoregressive methods on machine translation. **SunDAE** is **iterative denoising with training that matches inference** — unrolling denoising steps during training for better non-autoregressive text generation.