contrastive learning

Contrastive learning is a self-supervised learning approach that learns representations by pulling similar (positive) examples together in embedding space while pushing dissimilar (negative) examples apart, enabling powerful feature learning without labeled data. Core principle: maximize agreement between differently augmented views of the same data (positive pairs) while minimizing agreement with other examples (negative pairs). Loss function: InfoNCE (contrastive loss)—L = -log(exp(sim(z_i, z_j)/τ) / Σ_k exp(sim(z_i, z_k)/τ)), where z_i and z_j are embeddings of positive pair, z_k are negatives, sim is similarity (cosine), τ is temperature. Key components: (1) data augmentation (create positive pairs—crop, color jitter, blur for images), (2) encoder (neural network mapping inputs to embeddings), (3) projection head (MLP mapping embeddings to contrastive space), (4) contrastive loss (InfoNCE, NT-Xent). Influential methods: (1) SimCLR (simple framework with strong augmentations, large batch sizes), (2) MoCo (momentum contrast—queue of negatives, momentum encoder), (3) BYOL (bootstrap your own latent—no explicit negatives), (4) SimSiam (simple siamese networks—stop-gradient), (5) SwAV (clustering-based). Vision applications: pre-train on ImageNet (unlabeled), fine-tune on downstream tasks—achieves supervised performance with 1-10% of labels. NLP: sentence embeddings (SimCSE), language model pre-training. Advantages: (1) learns from unlabeled data (abundant), (2) learns general representations (transfer well), (3) robust features (invariant to augmentations). Challenges: (1) requires large batch sizes or memory banks (many negatives), (2) sensitive to augmentation choices, (3) computational cost (multiple forward passes per sample). Contrastive learning has become dominant self-supervised learning paradigm, enabling foundation models trained on massive unlabeled datasets.