contrastive predictive coding
**Contrastive Predictive Coding (CPC)** is a **self-supervised representation learning method that trains neural encoders by predicting future observations in latent space using contrastive objectives — maximizing mutual information between a compact context representation and future encoded observations while distinguishing true futures from random negative samples** — introduced by van den Oord et al. (DeepMind, 2018) as a unifying framework that simultaneously achieved state-of-the-art self-supervised representations for speech, images, text, and reinforcement learning, directly inspiring wav2vec, SimCLR, and the broader contrastive learning revolution.
**What Is CPC?**
- **Core Idea**: Learn representations that are maximally informative about the future by training a model to predict future latent codes from a context — without ever predicting raw pixels or audio waveforms.
- **Encoder**: Maps raw observations (audio frames, image patches, words) to latent representations z_t.
- **Autoregressive Context Model**: A recurrent network aggregates past representations into a context vector c_t, which summarizes the history up to time t.
- **Prediction**: Linear predictors W_k map context c_t to predicted future representations for k steps ahead: z_hat_{t+k} = W_k c_t.
- **InfoNCE Loss**: The model is trained to identify the true future z_{t+k} among N-1 randomly sampled "negative" representations from the same batch — a contrastive multi-class classification problem.
**Why Predict in Latent Space?**
- **Avoids Modeling Irrelevant Details**: Predicting raw waveforms or pixels is dominated by low-level statistics. Predicting latent codes focuses the model on semantically informative structure.
- **Slow Features**: Meaningful semantic content (speaker identity, object category, sentence meaning) changes more slowly than raw signal variations — latent prediction captures these slow features.
- **Mutual Information Bound**: The InfoNCE loss is a lower bound on I(z_{t+k}; c_t) — the mutual information between the context and the future. Maximizing InfoNCE maximizes predictive mutual information.
**Influence on Self-Supervised Learning**
| Method | How It Extends CPC |
|--------|--------------------|
| **wav2vec 2.0** | CPC applied to quantized speech codes — foundation of modern ASR |
| **SimCLR** | Drops temporal structure; applies contrastive prediction to augmented image pairs |
| **MoCo** | Momentum encoder + memory bank for large negative sets — CPC scaled for vision |
| **Data2Vec** | Generalizes CPC's predictive coding idea across speech, vision, and language |
| **CPC for RL (CURL, ATC)** | Applies contrastive coding to RL state representations |
**Applications**
- **Speech**: CPC representations transfer to phoneme detection, speaker verification, and ASR without any labeled data — demonstrating that temporal predictability captures phonetic structure.
- **Computer Vision**: Predicting spatial patches from context in images learns features competitive with fully supervised models.
- **Natural Language Processing**: Temporal CPC on sentences learns bidirectional contextual representations.
- **Reinforcement Learning**: CPC state encoders improve sample efficiency dramatically in pixel-observation RL tasks.
Contrastive Predictive Coding is **the self-supervised principle that the best representations are those that predict the future** — the insight that learning to forecast in latent space extracts the structural regularities of the world, producing representations that transfer broadly across downstream tasks without a single manual label.