multinomial diffusion

**Multinomial Diffusion** is a **discrete diffusion model where the forward process corrupts categorical data using a categorical (multinomial) noise distribution** — at each timestep, each token has a probability of being replaced by any other token in the vocabulary according to a multinomial transition matrix. **Multinomial Diffusion Details** - **Transition Matrix**: $q(x_t | x_{t-1}) = Cat(x_t; Q_t x_{t-1})$ — categorical distribution over vocabulary. - **Uniform Noise**: The simplest scheme transitions toward a uniform distribution over all tokens. - **Absorbing**: Alternative scheme transitions toward a single [MASK] token — absorbing state diffusion. - **Reverse**: $p_ heta(x_{t-1} | x_t) = Cat(x_{t-1}; pi_ heta(x_t, t))$ — neural network predicts clean token probabilities. **Why It Matters** - **Natural Fit**: Multinomial diffusion is mathematically natural for text, categorical features, and one-hot encoded data. - **D3PM**: Structured Denoising Diffusion Models (Austin et al., 2021) formalized multinomial and absorbing diffusion. - **Flexibility**: Different transition matrices enable different noise schedules — uniform, absorbing, or token-similarity-based. **Multinomial Diffusion** is **random token scrambling and unscrambling** — a discrete diffusion process using categorical transitions for generating text, molecules, and other categorical data.