Energy-Based Models (EBMs) is the probabilistic framework assigning energy values to configurations, where probability inversely proportional to energy — trainable via contrastive divergence or score matching to enable joint learning of generative and discriminative patterns.

Energy-Based Modeling Framework:

• Energy function: E(x) assigns scalar energy to each configuration x; lower energy → higher probability
• Unnormalized probability: p(x) ∝ exp(-E(x)); partition function Z = ∫exp(-E(x))dx often intractable
• Boltzmann distribution: statistical mechanics connection; energy models sample from Gibbs/Boltzmann distribution
• Inference: finding minimum-energy configuration (MAP inference); related to constraint satisfaction

Training via Contrastive Divergence:

• Contrastive divergence (CD): approximate maximum likelihood training without computing partition function
• Data distribution: positive phase collects samples from data; learning increases probability of data
• Model distribution: negative phase collects samples from model; learning decreases probability of model samples
• K-step CD: run K steps MCMC from data point; data samples naturally distributed; model samples biased but practical
• Practical approximation: CD-1 (single Gibbs step) often sufficient; reduces computational cost from intractable exact MLE

MCMC Sampling via Langevin Dynamics:

• Langevin dynamics: gradient-based MCMC sampling from energy function; iterative process: x_{t+1} = x_t - η∇E(x_t) + noise
• Gradient direction: move opposite to energy gradient (downhill in energy landscape); noise ensures Markov chain ergodicity
• Convergence: Langevin dynamics samples from exp(-E(x)) after sufficient iterations; enables efficient sampling
• Mixing time: number of steps to converge depends on energy landscape; sharp minima require more steps

Score Matching:

• Score function: ∇_x log p(x) is score; matching score equivalent to matching density without computing partition function
• Denoising score matching: add Gaussian noise to data; match denoised score; avoids manifold singularities
• Sliced score matching: project score onto random directions; reduces dimensionality and computational cost
• Score-based generative models: train score function; sample via reverse SDE (score-based diffusion models); related to EBMs

Joint EBM Architecture:

• Discriminative + generative: single energy function used for both classification and generation
• Discriminative application: conditional energy E(y|x); enables joint learning of class boundaries and data generation
• Hybrid learning: supervised loss + generative contrastive loss; improves both classification and generation
• Parameter sharing: single network learns both tasks; more parameter-efficient than separate models

EBM Applications:

• Anomaly detection: high-energy examples are anomalous; learned energy function detects out-of-distribution examples
• Image generation: sample via MCMC from learned energy function; slower than GANs but theoretically principled
• Structured prediction: energy incorporates constraints; inference finds satisfying assignments; useful for combinatorial problems
• Collaborative filtering: energy models user-item interactions; joint learning with side information

Connection to Denoising Diffusion Models:

• Score matching foundation: modern diffusion models train score function via score matching; equivalent to denoising objective
• Reverse process: sampling uses score (energy gradient); Langevin dynamics evolution generates samples
• Generative modeling: diffusion models successful application of score-based approach; practical and scalable

EBM Challenges:

• Sampling inefficiency: MCMC sampling slow compared to direct generation (GANs); limits practical application
• Evaluation difficulty: partition function intractable; evaluating likelihood challenging; no natural likelihood objective
• Scalability: contrastive divergence requires two phases (data + model); computational overhead
• Mode coverage: mode collapse possible if positive/negative phases don't mix well

Energy-based models provide principled probabilistic framework assigning energy to configurations — trainable without computing intractable partition functions via contrastive divergence or score matching for generation and discrimination.