discrete diffusion

**Discrete Diffusion** models are **generative models that apply the diffusion framework to discrete data (tokens, categories, graphs)** — instead of adding Gaussian noise to continuous values, discrete diffusion corrupts data by randomly replacing tokens with other tokens or a mask state, then learns to reverse this corruption process. **Discrete Diffusion Approach** - **Forward Process**: Gradually corrupt discrete tokens — replace with random tokens or [MASK] at increasing rates. - **Transition Matrix**: A categorical transition matrix $Q_t$ defines the corruption probabilities at each timestep. - **Absorbing State**: One variant uses an absorbing [MASK] state — tokens are progressively masked until all are masked. - **Reverse Process**: A neural network learns to predict the original tokens from corrupted sequences. **Why It Matters** - **Text Generation**: Enables non-autoregressive text generation using diffusion — competitive with autoregressive models. - **Molecules**: Discrete diffusion generates molecular graphs — atoms and bonds are discrete structures. - **Categorical Data**: Natural for any domain with categorical variables — proteins, music, code. **Discrete Diffusion** is **noise-and-denoise for categories** — extending the diffusion model framework from continuous data to discrete tokens and structures.