contrastive learning self supervised

**Contrastive Learning** is **the self-supervised representation learning paradigm where a model learns to distinguish between similar (positive) and dissimilar (negative) pairs of data augmentations — producing embeddings where semantically similar inputs are mapped nearby and dissimilar inputs are pushed apart, all without requiring human-annotated labels**. **Core Principles:** - **Positive Pairs**: two augmented views of the same image — random crop, color jitter, Gaussian blur, horizontal flip applied independently to create two correlated views (x_i, x_j) that should have similar embeddings - **Negative Pairs**: augmented views from different images — all other images in the mini-batch serve as negatives; more negatives provide better coverage of the representation space but require more memory - **InfoNCE Loss**: L = -log(exp(sim(z_i,z_j)/τ) / Σ_k exp(sim(z_i,z_k)/τ)) — maximizes agreement between positive pair relative to all negatives; temperature τ controls how hard negatives are emphasized (typical τ=0.07-0.5) - **Projection Head**: non-linear MLP applied after the backbone encoder — maps representations to a space where contrastive loss is applied; the pre-projection representations transfer better to downstream tasks **Major Frameworks:** - **SimCLR**: end-to-end contrastive learning within a mini-batch — requires large batch sizes (4096-8192) to provide sufficient negatives; uses NT-Xent loss with cosine similarity; simple but compute-intensive - **MoCo (Momentum Contrast)**: maintains a queue of negatives from recent mini-batches — momentum-updated encoder produces consistent negative representations; decouples negative count from batch size enabling smaller batches (256) - **BYOL (Bootstrap Your Own Latent)**: eliminates negative pairs entirely — online network predicts the representation of a target network (momentum-updated); avoids mode collapse through asymmetric architecture and momentum update - **SwAV (Swapping Assignments)**: assigns augmented views to learned prototype clusters — enforces consistency: view 1's assignment should match view 2's assignment; combines contrastive learning with clustering for multi-crop efficiency **Training and Transfer:** - **Pre-Training Scale**: competitive contrastive learning requires 200-1000 training epochs on ImageNet — compared to 90 epochs for supervised training; long training compensates for weaker per-sample supervision - **Linear Evaluation Protocol**: freeze pre-trained backbone, train only a linear classifier on top — standard benchmark for representation quality; SimCLR achieves 76.5%, supervised achieves 78.2% on ImageNet - **Fine-Tuning Transfer**: pre-trained representations fine-tuned on downstream tasks — contrastive pre-training often outperforms supervised pre-training for transfer learning, especially with limited labeled data (10-100× improvement at 1% label fraction) - **Multi-Modal Contrastive (CLIP)**: contrasts image-text pairs from internet data — learns aligned vision-language representations enabling zero-shot classification; 400M image-text pairs produces representations that transfer broadly without fine-tuning **Contrastive learning has fundamentally changed the deep learning landscape by demonstrating that high-quality visual representations can be learned without any human labels — enabling AI systems trained on vast unlabeled data to match or exceed the performance of fully supervised methods.**