energy based models ebm

**Energy-Based Models (EBMs)** are **a general class of generative models that define a probability distribution over data by assigning a scalar energy value to each input configuration, with lower energy corresponding to higher probability** — offering a flexible, unnormalized modeling framework where the energy function can be parameterized by arbitrary neural networks without the architectural constraints imposed by normalizing flows or the training instability of GANs. **Mathematical Foundation:** - **Energy Function**: A learned function E_theta(x) maps each data point x to a scalar energy value; the model does not require E to have any specific structure beyond being differentiable with respect to its parameters - **Boltzmann Distribution**: The probability density is defined as p_theta(x) = exp(-E_theta(x)) / Z_theta, where Z_theta is the partition function (normalizing constant) obtained by integrating exp(-E) over all possible inputs - **Intractable Partition Function**: Computing Z_theta requires integrating over the entire data space, which is infeasible for high-dimensional inputs — making maximum likelihood training challenging and motivating approximate training methods - **Free Energy**: For models with latent variables, the free energy marginalizes over latent configurations: F(x) = -log(sum_h exp(-E(x, h))), connecting EBMs to traditional probabilistic graphical models **Training Methods:** - **Contrastive Divergence (CD)**: Approximate the gradient of the log-likelihood by running k steps of MCMC (typically Gibbs sampling) starting from data points; CD-1 uses a single step and was instrumental in training Restricted Boltzmann Machines - **Persistent Contrastive Divergence (PCD)**: Maintain persistent MCMC chains across training iterations rather than reinitializing from data, producing better gradient estimates at the cost of maintaining a replay buffer of negative samples - **Score Matching**: Minimize the squared difference between the model's score function (gradient of log-density) and the data score, avoiding partition function computation entirely; equivalent to denoising score matching when noise is added to data - **Noise Contrastive Estimation (NCE)**: Train a binary classifier to distinguish data from noise samples, implicitly learning the energy function as the log-ratio of data to noise density - **Sliced Score Matching**: Project the score matching objective onto random directions, reducing computational cost from computing the full Hessian trace to evaluating directional derivatives - **Denoising Score Matching (DSM)**: Perturb data with known noise and train the model to estimate the score of the noised distribution — directly connected to the training of diffusion models **Sampling from EBMs:** - **Langevin Dynamics (SGLD)**: Initialize samples from noise, then iteratively update them by following the gradient of the log-density plus Gaussian noise: x_t+1 = x_t + (step/2) * grad_x log p(x_t) + sqrt(step) * noise - **Hamiltonian Monte Carlo (HMC)**: Augment the state with momentum variables and simulate Hamiltonian dynamics to produce distant, low-autocorrelation samples - **Replay Buffer**: Maintain a buffer of previously generated samples and use them to initialize SGLD chains, dramatically reducing the mixing time needed for high-quality samples - **Short-Run MCMC**: Use very few MCMC steps (10–100) for each sample, accepting that samples are not fully converged but sufficient for training signal - **Amortized Sampling**: Train a separate generator network to produce approximate samples, which are then refined with a few MCMC steps — combining the speed of amortized inference with EBM flexibility **Connections to Other Generative Models:** - **Diffusion Models**: Score-based diffusion models can be viewed as EBMs trained at multiple noise levels, with Langevin dynamics providing the sampling mechanism — DSM is their primary training objective - **GANs**: The discriminator in a GAN can be interpreted as an energy function, and some EBM training methods resemble adversarial training - **Normalizing Flows**: Flows provide tractable density evaluation but with architectural constraints; EBMs trade tractable density for maximal architectural flexibility - **Variational Autoencoders**: VAEs optimize a lower bound on log-likelihood with amortized inference; EBMs can use MCMC for more accurate but slower posterior estimation **Applications:** - **Compositional Generation**: Energy functions naturally compose through addition (product of experts), enabling modular generation where multiple EBMs controlling different attributes combine during sampling - **Out-of-Distribution Detection**: Use energy values as confidence scores — in-distribution data receives low energy, out-of-distribution inputs receive high energy - **Classifier-Free Guidance**: The guidance mechanism in modern diffusion models is interpretable as composing conditional and unconditional energy functions - **Protein Structure Prediction**: Model the energy landscape of protein conformations, with low-energy states corresponding to stable folded structures Energy-based models provide **the most general and flexible framework for probabilistic generative modeling — where the freedom to define arbitrary energy landscapes comes at the cost of intractable normalization, motivating a rich ecosystem of approximate training and sampling methods that have profoundly influenced the development of modern diffusion models and score-based generative approaches**.