noise contrastive estimation for ebms
**Noise Contrastive Estimation (NCE) for Energy-Based Models** is a **training technique that replaces the intractable maximum likelihood objective for Energy-Based Models with a binary classification problem** — distinguishing real data samples from synthetic "noise" samples drawn from a known distribution, implicitly estimating the unnormalized log-density ratio between the data and noise distributions without computing the intractable partition function, enabling practical EBM training for continuous high-dimensional data.
**The Fundamental EBM Training Problem**
Energy-Based Models define an unnormalized density:
p_θ(x) = exp(-E_θ(x)) / Z(θ)
where E_θ(x) is the learned energy function and Z(θ) = ∫ exp(-E_θ(x)) dx is the partition function.
Maximum likelihood training requires computing ∇_θ log Z(θ), which equals:
∇_θ log Z = E_{x~p_θ}[−∇_θ E_θ(x)]
This expectation is over the model distribution p_θ — requiring MCMC sampling from the current model at every gradient step. MCMC mixing is slow in high dimensions, making naive maximum likelihood training impractical for complex distributions.
**The NCE Solution**
NCE (Gutmann and Hyvärinen, 2010) reformulates density estimation as binary classification:
Given: data samples from p_data(x) (positive class) and noise samples from a fixed, known q(x) (negative class).
Train a classifier h_θ(x) = P(class = data | x) to distinguish the two:
h_θ(x) = p_θ(x) / [p_θ(x) + ν · q(x)]
where ν is the noise-to-data ratio. When optimized with binary cross-entropy:
L_NCE(θ) = E_{x~p_data}[log h_θ(x)] + ν · E_{x~q}[log(1 - h_θ(x))]
The optimal classifier satisfies h*(x) = p_data(x) / [p_data(x) + ν · q(x)], which means the classifier implicitly estimates the log-density ratio log[p_data(x) / q(x)].
If we parametrize h_θ such that the log-ratio equals an explicit energy function:
log h_θ(x) - log(1 - h_θ(x)) = log p_data(x) - log q(x) ≈ -E_θ(x) - log Z_q
then training the classifier corresponds to learning the energy function up to a constant (the log partition function of q, which is known since q is known).
**Choice of Noise Distribution**
The noise distribution q(x) is the critical design choice:
| Noise Distribution | Properties | Performance |
|-------------------|------------|-------------|
| **Gaussian** | Simple, easy to sample | Poor if data is far from Gaussian |
| **Uniform** | Very simple | Ineffective for concentrated data |
| **Product of marginals** | Destroys correlations, simple | Captures marginals but not structure |
| **Flow model** | Adaptively approximates data | Expensive to sample, but NCE converges faster |
| **Replay buffer (IGEBM)** | Past model samples | Self-competitive, approaches data distribution |
**Connection to Maximum Likelihood and Contrastive Divergence**
NCE becomes exact maximum likelihood as ν → ∞ and q → p_θ (the noise approaches the model itself). This is the connection to contrastive divergence — when the noise distribution is the current model, NCE reduces to a single-step MCMC gradient estimator.
**Connection to GANs**
NCE bears a deep structural similarity to GAN training:
- GAN discriminator: distinguishes real from generated samples
- NCE classifier: distinguishes real from noise samples
The key difference: NCE uses a fixed, external noise distribution, while GANs simultaneously train the generator to fool the discriminator. NCE is simpler (no minimax optimization) but cannot adapt the noise to hard negatives.
**Modern Applications**
**Contrastive Language-Image Pre-training (CLIP)**: NCE is the conceptual foundation of contrastive learning objectives. InfoNCE (Oord et al., 2018) applies NCE to representation learning: positive pairs (image, matching caption) vs. negative pairs (image, random caption) — learning representations where matching pairs have lower energy.
**Language model vocabulary learning**: NCE avoids the O(vocabulary size) softmax computation in language models, replacing it with a small negative sample set for efficient large-vocabulary training.
**Partition function estimation**: Given a trained EBM, NCE with a tractable reference distribution provides unbiased estimates of Z(θ) for likelihood evaluation.