energy-based models

**Energy-Based Models (EBMs)** are a **class of generative models that define a probability distribution through an energy function** — $p_ heta(x) = exp(-E_ heta(x)) / Z$ where lower energy corresponds to higher probability, and the model learns to assign low energy to data-like inputs. **Key Concepts** - **Energy Function**: $E_ heta(x)$ is a neural network mapping inputs to a scalar energy value. - **Partition Function**: $Z = int exp(-E_ heta(x)) dx$ — intractable normalization constant. - **Sampling**: MCMC methods (Langevin dynamics, HMC) generate samples by following the energy gradient. - **Training**: Contrastive divergence, score matching, or noise contrastive estimation (NCE) avoid computing $Z$. **Why It Matters** - **Flexibility**: EBMs can model arbitrary distributions without architectural constraints (no decoder, no normalizing flow). - **Composability**: Multiple EBMs can be combined by adding energies — $E_{joint} = E_1 + E_2$. - **Discriminative + Generative**: The same energy function can be used for both classification and generation (JEM). **EBMs** are **learning an energy landscape** — defining probability through energy where likely configurations sit in low-energy valleys.