self supervised learning

**Self-Supervised Learning (SSL)** is the **representation learning paradigm that trains neural networks on massive unlabeled datasets by defining proxy objectives — contrastive, predictive, or self-distillation tasks — that force the model to learn rich, transferable visual and textual features without a single human annotation**. **Why SSL Changed the Game** Labeling images at ImageNet scale costs hundreds of thousands of dollars and months of annotator time. SSL methods extract comparable or superior feature quality from raw, uncurated data, decoupling model capability from labeling budgets. DINO's self-supervised ViT features contain emergent object segmentation maps that no supervised model was ever explicitly taught. **The Three Major Families** - **Contrastive (SimCLR, MoCo)**: Two augmented views of the same image are pulled together in embedding space while views from different images are pushed apart. SimCLR requires very large batch sizes (4096+) to supply enough negative examples; MoCo maintains a momentum-updated queue of negatives to decouple batch size from negative count. - **Non-Contrastive (BYOL, VICReg)**: BYOL uses an online network that predicts the output of a slowly-updated momentum teacher network. No negative pairs are needed. Collapse prevention relies on the asymmetric architecture (stop-gradient on the teacher) rather than explicit repulsion terms. - **Self-Distillation (DINO, DINOv2)**: A student network is trained to match the softmax probability distribution of a momentum teacher across different crops of the same image. The teacher's centering and sharpening operations prevent mode collapse without negatives. **Critical Hyperparameters** - **Augmentation Policy**: Random resized crop, color jitter, Gaussian blur, and solarization define the invariances the SSL objective will learn. Wrong augmentations teach wrong invariances — aggressive color jitter on a pathology dataset would destroy diagnostically critical color information. - **Projection Head**: A 2-3 layer MLP maps the backbone features into a lower-dimensional space where the SSL loss is computed. Critically, this projection head is discarded after pretraining; only the backbone transfers. - **Temperature**: Controls the sharpness of the contrastive or distillation distribution. Too low produces gradient instability and collapse; too high washes out informative structure. **Transfer Quality Evaluation** The gold standard is linear probing — freezing the SSL backbone and training only a single linear classifier on a downstream task with limited labels. Competitive SSL methods match or exceed supervised ImageNet pretraining on 20+ downstream benchmarks across detection, segmentation, and classification. Self-Supervised Learning is **the foundation of modern visual AI at scale** — eliminating the annotation bottleneck that previously gated the quality of every computer vision model on the budget available for manual labeling.