contrastive learning

**Contrastive Learning / Self-Supervised Learning** — training models to learn useful representations from unlabeled data by contrasting similar (positive) and dissimilar (negative) pairs. **Core Idea** - Create two augmented views of the same image (positive pair) - Pull their representations together in embedding space - Push representations of different images apart - No labels needed — the augmentation defines the learning signal **Key Methods** - **SimCLR**: Simple framework. Augment → encode → project → contrastive loss (InfoNCE). Needs large batches (4096+) - **MoCo (Momentum Contrast)**: Maintains a momentum-updated queue of negatives. Works with normal batch sizes - **BYOL (Bootstrap Your Own Latent)**: No negatives at all — uses a momentum target network. Surprisingly effective - **DINO/DINOv2**: Self-distillation with no labels. Produces exceptional image features - **MAE (Masked Autoencoder)**: Mask 75% of image patches → reconstruct. Vision analog of BERT **Why It Matters** - Labeled data is expensive and limited - Self-supervised models trained on billions of unlabeled images learn better features than supervised training - Foundation models (CLIP, DINOv2) are self/weakly supervised **Performance** - DINOv2 features match or beat supervised features on downstream tasks - Self-supervised pretraining is now the default for large vision models **Self-supervised learning** is how modern AI escapes the bottleneck of labeled data.