contrastive divergence

**Contrastive Divergence (CD)** is a **training algorithm for energy-based models that approximates the gradient of the log-likelihood** — using short-run MCMC (typically just 1 step of Gibbs sampling or Langevin dynamics) instead of running the chain to equilibrium, making EBM training practical. **How CD Works** - **Positive Phase**: Compute the gradient of the energy at data points (easy: just backprop through $E_ heta(x_{data})$). - **Negative Phase**: Run $k$ steps of MCMC from the data to get approximate model samples. - **Gradient**: $ abla_ heta log p approx - abla_ heta E(x_{data}) + abla_ heta E(x_{MCMC})$ (push down data energy, push up sample energy). - **CD-k**: $k$ is the number of MCMC steps (CD-1 is most common — just 1 step). **Why It Matters** - **Practical Training**: CD makes EBM training feasible by avoiding the need for converged MCMC chains. - **RBMs**: CD was the breakthrough that made training Restricted Boltzmann Machines practical (Hinton, 2002). - **Bias**: CD introduces bias (unconverged MCMC), but works well in practice for many EBMs. **Contrastive Divergence** is **the shortcut for EBM training** — using a few MCMC steps instead of full equilibration to approximate the intractable gradient.