score based generative model
**Score-Based Generative Models** are **generative models that learn the score function (gradient of the log probability density) ∇_x log p(x) across multiple noise levels**, then generate samples by following the learned score through a reverse-time stochastic differential equation (SDE) or equivalent ODE — unifying denoising diffusion models and score matching under a continuous-time framework.
**The Score Function**: For a data distribution p(x), the score is the vector field s(x) = ∇_x log p(x). The score points in the direction of steepest increase of probability density. If we know the score everywhere, we can generate samples by starting from random noise and following the score (Langevin dynamics): x_{t+1} = x_t + ε/2 · s(x_t) + √ε · z where z ~ N(0,I).
**The Problem with Raw Data**: Score estimation directly on clean data fails because the score is undefined in low-density regions (where log p → -∞) and data lies on lower-dimensional manifolds in high-dimensional space. Solution: **add noise at multiple scales** to smooth the data distribution, learn scores for each noise level, and then generate by gradually denoising.
**SDE Framework** (Song et al., 2021):
| Component | Forward SDE | Reverse SDE |
|-----------|------------|------------|
| Equation | dx = f(x,t)dt + g(t)dw | dx = [f(x,t) - g(t)²∇_x log p_t(x)]dt + g(t)dw̄ |
| Direction | Data → Noise | Noise → Data |
| Time | t: 0 → T | t: T → 0 |
| Purpose | Define noise process | Generate samples |
The forward SDE gradually adds noise, converting data into a simple prior (Gaussian). The reverse SDE generates samples by removing noise, requiring only the score ∇_x log p_t(x) at each noise level t.
**Connection to DDPM**: Denoising Diffusion Probabilistic Models (DDPM) are a discrete-time special case where the forward SDE is a Variance-Preserving (VP) process: dx = -½β(t)x dt + √β(t) dw. The denoising network ε_θ(x_t, t) is related to the score by: s_θ(x_t, t) = -ε_θ(x_t, t) / σ(t). Training with the simple MSE loss ‖ε - ε_θ(x_t, t)‖² is equivalent to denoising score matching.
**Probability Flow ODE**: For any SDE, there exists a deterministic ODE whose trajectories have the same marginal distributions: dx = [f(x,t) - ½g(t)²∇_x log p_t(x)]dt. This ODE enables: **exact likelihood computation** (via the change of variables formula); **deterministic sampling** (same noise → same sample, enabling interpolation); and **faster sampling** (ODE solvers can use larger steps than SDE solvers).
**Sampling Speed**: The major practical challenge. Full SDE sampling requires ~1000 steps. Acceleration methods: **DDIM** (deterministic ODE-based sampler, 50-250 steps); **DPM-Solver** (exponential integrator for the diffusion ODE, 10-20 steps); **Consistency Models** (distill multi-step process into 1-2 step generation); and **progressive distillation** (iteratively halve the number of steps).
**Score-based generative models provide the most mathematically rigorous framework for diffusion-based generation — connecting deep learning to stochastic calculus and enabling principled trade-offs between sample quality, diversity, speed, and exact likelihood computation.**