score-based generative models
**Score-Based Generative Models** are a class of generative models that learn the score function ∇_x log p(x)—the gradient of the log-probability density with respect to the data—rather than the density itself, then use the learned score to generate samples through iterative score-based sampling procedures such as Langevin dynamics. This approach avoids the normalization constant computation that makes direct density modeling intractable for complex, high-dimensional distributions.
**Why Score-Based Generative Models Matter in AI/ML:**
Score-based models provide **state-of-the-art generative quality** by sidestepping the fundamental challenge of normalizing constant computation, leveraging the fact that the score function contains all the information needed for sampling without requiring a tractable partition function.
• **Score function** — The score ∇_x log p(x) is a vector field pointing in the direction of increasing log-density at every point in data space; following this gradient (with noise) from any starting point converges to samples from p(x) via Langevin dynamics
• **Score matching training** — Directly minimizing E[||s_θ(x) - ∇_x log p(x)||²] is intractable (requires knowing the true score); denoising score matching instead trains on noisy data: s_θ(x̃) ≈ ∇_{x̃} log p(x̃|x) = -(x̃-x)/σ², which is tractable and consistent
• **Multi-scale noise perturbation** — Score estimation is inaccurate in low-density regions (few training examples); adding noise at multiple scales (σ₁ > σ₂ > ... > σ_N) fills in low-density regions and creates a sequence of score functions from coarse to fine
• **Connection to diffusion** — Score-based models and denoising diffusion probabilistic models (DDPMs) are equivalent formulations: the DDPM denoiser ε_θ is related to the score by s_θ(x_t, t) = -ε_θ(x_t, t)/σ_t; this unification bridges the two research communities
• **SDE formulation** — Song et al. unified score-based and diffusion models through stochastic differential equations (SDEs): the forward SDE gradually adds noise, and the reverse-time SDE (requiring the score function) generates samples by denoising
| Component | Role | Implementation |
|-----------|------|---------------|
| Score Network s_θ | Estimates ∇_x log p(x) | U-Net, Transformer (time-conditioned) |
| Noise Schedule | Multi-scale perturbation | σ₁ > σ₂ > ... > σ_N or continuous σ(t) |
| Training Loss | Denoising score matching | E[||s_θ(x+σε) + ε/σ||²] |
| Sampling | Reverse-time SDE/ODE | Langevin dynamics, predictor-corrector |
| SDE Forward | dx = f(x,t)dt + g(t)dw | VP-SDE, VE-SDE, sub-VP-SDE |
| SDE Reverse | dx = [f - g²∇log p]dt + gdw̄ | Score-guided denoising |
**Score-based generative models represent a paradigm shift in generative modeling by learning the gradient of the log-density rather than the density itself, unifying with diffusion models through the SDE framework and achieving state-of-the-art image generation quality by sidestepping normalization constant computation while enabling flexible, iterative sampling through learned score functions.**