noise contrastive estimation

**Noise Contrastive Estimation (NCE)** is a **statistical estimation technique that trains a model to distinguish real data from artificially generated noise** — by converting an unsupervised density estimation problem into a supervised binary classification problem. **What Is NCE?** - **Idea**: Instead of computing the intractable normalization constant $Z$ of an energy-based model, train a classifier to distinguish "real" data from "noise" samples drawn from a known distribution. - **Loss**: Binary cross-entropy between real data (label=1) and noise data (label=0). - **Result**: The model learns the log-ratio of data density to noise density, which is proportional to the unnormalized log-likelihood. **Why It Matters** - **Foundation**: Inspired InfoNCE (the multi-class extension used in contrastive learning). - **Language Models**: Word2Vec's negative sampling is a simplified form of NCE. - **Efficiency**: Avoids computing the partition function $Z$ (which requires summing over all possible outputs). **NCE** is **learning by telling real from fake** — a powerful trick that converts intractable density estimation into simple classification.