annealed langevin dynamics
**Annealed Langevin Dynamics** is a multi-scale sampling technique for score-based generative models that generates samples by running Langevin dynamics at a sequence of decreasing noise levels, starting from a highly noisy distribution (easy to sample from and mix between modes) and gradually transitioning to the clean data distribution. At each noise level σ_l, sampling uses the noise-conditional score estimate s_θ(x, σ_l) learned via denoising score matching.
**Why Annealed Langevin Dynamics Matters in AI/ML:**
Annealed Langevin dynamics solves the **multi-modality and low-density region problems** that prevent standard Langevin dynamics from generating high-quality samples, enabling the first practical score-based generative models (NCSN) that rivaled GANs in image generation quality.
• **Multi-scale noise schedule** — A geometric sequence of noise levels σ₁ > σ₂ > ... > σ_L (e.g., σ₁=50, σ_L=0.01) defines the annealing schedule; at σ₁, the noisy data distribution is nearly Gaussian (easy to traverse); at σ_L, it closely approximates the clean data distribution
• **Mode traversal at high noise** — Large noise levels smooth out the data distribution, filling valleys between modes and enabling Langevin dynamics to move freely between modes that would be separated by energy barriers at low noise levels
• **Progressive refinement** — Starting from coarse structure (high noise) and progressively adding detail (low noise) mirrors a coarse-to-fine generation process: global structure is determined first, then textures and fine details are refined in later stages
• **Per-level score estimation** — The score network s_θ(x, σ) is conditioned on the noise level, providing appropriate gradients at each scale: high-noise scores capture global structure, low-noise scores capture fine details
• **NCSN (Noise Conditional Score Network)** — The original model (Song & Ermon 2019) that demonstrated annealed Langevin dynamics for image generation, training a single noise-conditional score network and sampling through the annealing procedure
| Noise Level | Distribution Character | Langevin Behavior | Generation Role |
|------------|----------------------|-------------------|----------------|
| σ₁ (largest) | Near-Gaussian, unimodal | Fast mixing, mode exploration | Global structure |
| σ₂-σ_{L/3} | Smoothed, merged modes | Cross-mode transitions | Coarse layout |
| σ_{L/3}-σ_{2L/3} | Multi-modal, clearer modes | Mode-local refinement | Mid-level features |
| σ_L (smallest) | Near-clean data | Fine-tuning, high-frequency | Textures, details |
| Steps per Level | T₁ = T₂ = ... = T | Equal or proportional to σ² | Convergence time |
**Annealed Langevin dynamics is the breakthrough sampling technique that made score-based generative models practical by addressing the fundamental challenges of multi-modality and sparse data regions through a hierarchical, coarse-to-fine noise annealing procedure that progressively transforms random noise into high-quality data samples guided by learned score functions at each noise level.**