score matching for ebms

**Score Matching** is a **training method for energy-based models that avoids computing the intractable partition function** — by matching the gradient (score) of the model's log-density to the gradient of the data distribution, which does not require normalization. **How Score Matching Works** - **Score**: The score function is $s_ heta(x) = abla_x log p_ heta(x) = - abla_x E_ heta(x)$ (gradient of energy). - **Objective**: Minimize $mathbb{E}_{p_{data}}[|s_ heta(x) - abla_x log p_{data}(x)|^2]$. - **Integration by Parts**: The unknown $ abla_x log p_{data}$ can be eliminated, giving: $mathbb{E}_{p_{data}}[ ext{tr}( abla_x s_ heta) + frac{1}{2}|s_ heta|^2]$. - **Denoising Score Matching**: An equivalent objective that matches the score of the noise-perturbed distribution. **Why It Matters** - **No Partition Function**: Score matching completely avoids the intractable normalization problem. - **Diffusion Models**: Modern diffusion models (DDPM, SDE-based) are trained with denoising score matching. - **Theoretically Sound**: Score matching is consistent — the optimal model has the correct data score. **Score Matching** is **learning gradients instead of densities** — training EBMs by matching the direction of steepest probability increase without computing $Z$.