absorbing state diffusion
**Absorbing State Diffusion** for text is a diffusion approach where **tokens gradually transition toward a special mask token (absorbing state)** — providing a natural discrete diffusion process where the forward process masks tokens with increasing probability and the reverse process learns to unmask, connecting diffusion models to masked language modeling like BERT.
**What Is Absorbing State Diffusion?**
- **Definition**: Diffusion process where tokens transition to [MASK] token (absorbing state).
- **Forward**: Tokens randomly replaced with [MASK] with increasing probability over time.
- **Reverse**: Model learns to predict original tokens from partially masked sequences.
- **Key Insight**: Masking is natural discrete corruption process.
**Why Absorbing State Diffusion?**
- **Natural for Discrete Data**: Masking is intuitive corruption for text.
- **Connection to BERT**: Leverages masked language modeling insights.
- **Simpler Than Continuous**: No embedding/projection complications.
- **Interpretable**: Easy to understand forward and reverse processes.
- **Effective**: Competitive with other discrete diffusion approaches.
**How It Works**
**Forward Process (Masking)**:
- **Start**: Clean text sequence x_0 = [token_1, token_2, ..., token_n].
- **Step t**: Each token has probability q(t) of being [MASK].
- **Schedule**: q(t) increases from 0 to 1 as t goes from 0 to T.
- **End**: x_T is fully masked [MASK, MASK, ..., MASK].
**Transition Probabilities**:
```
P(x_t = [MASK] | x_{t-1} = token) = β_t
P(x_t = token | x_{t-1} = token) = 1 - β_t
P(x_t = token | x_{t-1} = [MASK]) = 0 (absorbing!)
```
- **Absorbing**: Once masked, stays masked (can't unmask in forward process).
- **Schedule**: β_t defines masking rate at each step.
**Reverse Process (Unmasking)**:
- **Start**: Fully masked sequence x_T.
- **Model**: Transformer predicts original tokens from masked sequence.
- **Input**: Partially masked sequence + timestep t.
- **Output**: Probability distribution over tokens for each [MASK] position.
- **Sampling**: Sample tokens from predicted distribution, gradually unmask.
**Connection to BERT**
**Similarities**:
- **Masking**: Both use [MASK] token as corruption.
- **Prediction**: Both predict original tokens from masked context.
- **Bidirectional**: Both use bidirectional context for prediction.
**Differences**:
- **BERT**: Single masking level (15% typically), single prediction step.
- **Diffusion**: Multiple masking levels, iterative unmasking over T steps.
- **BERT**: Trained for representation learning.
- **Diffusion**: Trained for generation.
**Insight**: Absorbing state diffusion generalizes BERT to iterative generation.
**Training**
**Objective**:
- **Loss**: Cross-entropy between predicted and true tokens at masked positions.
- **Sampling**: Sample timestep t, mask according to schedule, predict original.
- **Optimization**: Standard supervised learning, no adversarial training.
**Training Algorithm**:
```
1. Sample clean sequence x_0 from dataset
2. Sample timestep t ~ Uniform(1, T)
3. Mask tokens according to schedule q(t)
4. Model predicts original tokens from masked sequence
5. Compute cross-entropy loss on masked positions
6. Backpropagate and update model
```
**Masking Schedule**:
- **Linear**: q(t) = t/T (uniform masking rate increase).
- **Cosine**: q(t) = cos²(πt/2T) (slower at start, faster at end).
- **Tuning**: Schedule affects generation quality, requires tuning.
**Generation (Sampling)**
**Iterative Unmasking**:
```
1. Start with fully masked sequence x_T = [MASK, ..., MASK]
2. For t = T down to 1:
a. Model predicts token probabilities for each [MASK]
b. Sample tokens from predicted distributions
c. Unmask some positions (according to schedule)
d. Keep other positions masked for next iteration
3. Final x_0 is generated text
```
**Unmasking Strategy**:
- **Confidence-Based**: Unmask positions with highest prediction confidence.
- **Random**: Randomly select positions to unmask.
- **Scheduled**: Unmask fixed fraction at each step.
**Temperature**:
- **Sampling**: Use temperature to control randomness.
- **Low Temperature**: More deterministic, higher quality.
- **High Temperature**: More diverse, more creative.
**Advantages**
**Natural Discrete Process**:
- **No Embedding**: No need to embed to continuous space.
- **No Projection**: No projection back to discrete tokens.
- **Interpretable**: Masking and unmasking are intuitive.
**Leverages BERT Insights**:
- **Pretrained Models**: Can initialize from BERT-like models.
- **Masked LM**: Builds on well-understood masked language modeling.
- **Transfer Learning**: Leverage existing masked LM research.
**Flexible Generation**:
- **Infilling**: Naturally handles filling masked spans.
- **Partial Generation**: Can fix some tokens, generate others.
- **Iterative Refinement**: Multiple passes improve quality.
**Controllable**:
- **Guidance**: Easy to apply constraints during unmasking.
- **Conditional**: Condition on various signals.
- **Editing**: Modify specific parts while keeping others.
**Limitations**
**Multiple Steps Required**:
- **Slow**: Requires T forward passes (typically T=50-1000).
- **Latency**: Higher latency than single autoregressive pass.
- **Trade-Off**: Quality vs. speed.
**Unmasking Order**:
- **Challenge**: Optimal unmasking order unclear.
- **Heuristics**: Confidence-based works but not optimal.
- **Impact**: Order affects generation quality.
**Long-Range Dependencies**:
- **Challenge**: Iterative unmasking may struggle with long-range coherence.
- **Autoregressive Advantage**: Left-to-right maintains coherence naturally.
- **Mitigation**: Careful schedule, more steps.
**Examples & Implementations**
**D3PM (Discrete Denoising Diffusion Probabilistic Models)**:
- **Approach**: Absorbing state diffusion for discrete data.
- **Application**: Text, images, graphs.
- **Performance**: Competitive with autoregressive on some tasks.
**MDLM (Masked Diffusion Language Model)**:
- **Approach**: Absorbing state diffusion specifically for language.
- **Connection**: Explicit connection to masked language modeling.
- **Performance**: Strong results on text generation benchmarks.
**Applications**
**Text Infilling**:
- **Task**: Fill in missing parts of text.
- **Advantage**: Naturally handles arbitrary masked spans.
- **Use Case**: Document completion, story writing.
**Controlled Generation**:
- **Task**: Generate text with constraints.
- **Advantage**: Easy to fix certain tokens, generate others.
- **Use Case**: Template filling, constrained generation.
**Text Editing**:
- **Task**: Modify specific parts of text.
- **Advantage**: Mask regions to edit, unmask with new content.
- **Use Case**: Paraphrasing, style transfer, improvement.
**Tools & Resources**
- **Research Papers**: D3PM, MDLM papers and code.
- **Implementations**: PyTorch/JAX implementations on GitHub.
- **Experimental**: Not yet in production frameworks.
Absorbing State Diffusion is **a promising approach for discrete diffusion** — by using masking as the corruption process, it provides a natural, interpretable way to apply diffusion to text that connects to successful masked language modeling, offering advantages in infilling, editing, and controllable generation while remaining simpler than continuous embedding approaches.