Contrastive Self-Supervised Learning is the unsupervised learning framework where models distinguish between augmented views of same sample (positive pairs) versus different samples (negative pairs) — learning rich visual representations rivaling supervised pretraining without labeled data.

Contrastive Learning Objective:

• Positive pairs: two augmented versions of same image; should have similar embeddings
• Negative pairs: augmentations of different images; should have dissimilar embeddings
• Contrastive loss: minimize distance for positives; maximize distance for negatives
• Unsupervised signal: no labels required; augmentation-induced variance provides learning signal
• Representation quality: learned representations effectively capture visual structure and semantic information

NT-Xent Loss (Normalized Temperature-Scaled Cross Entropy):

• Softmax contrast: normalize similarity scores; apply softmax and cross-entropy loss
• NT-Xent formulation: loss = -log[exp(sim(z_i, z_j)/τ) / ∑_k exp(sim(z_i, z_k)/τ)]
• Temperature parameter: τ controls distribution sharpness; τ = 0.07 typical; smaller τ → harder negatives
• Similarity metric: usually cosine similarity between normalized embeddings
• Batch as negatives: positive pair from single image; 2N-2 negatives from other batch samples

SimCLR Framework:

• Large batch size: 4096 samples typical; large batch provides diverse negatives
• Strong augmentation: color jitter, random crops, Gaussian blur; augmentation strength crucial
• Non-linear projection head: two-layer MLP with hidden dimension larger than output; improves downstream performance
• Contrastive training: large batch essential; 10x batch → 30% performance improvement
• Downstream fine-tuning: linear evaluation on frozen representations; evaluate transfer quality

Momentum Contrast (MoCo):

• Queue mechanism: maintain queue of previous embeddings; large dictionary without large batch
• Momentum encoder: slowly updated copy of main encoder via momentum (exponential moving average)
• Key advantage: decouples dictionary size from batch size; enables large dictionaries with manageable batch sizes
• MoCo variants: MoCo-v2 improves augmentations/projections; MoCo-v3 removes momentum encoder

Contrastive Learning Variants:

• BYOL (Bootstrap Your Own Latent): no negative pairs; momentum encoder and online networks; surprising finding
• SimSiam: simplified BYOL; just stop-gradient; shows importance of asymmetric architecture
• SwAV: online clustering and contrastive learning; cluster centroids provide self-labels
• DenseCL: dense prediction in contrastive learning; helps downstream dense prediction tasks

Representation Learning Insights:

• Invariance to augmentation: learned representation invariant to geometric/color transforms; semantic-preserving
• Feature reuse: representations learned via contrastive learning transfer well to downstream tasks
• Self-supervised equivalence: contrastive learning without labels approximates supervised learning quality
• Scaling with model size: larger models benefit from contrastive learning; improve supervised baselines

Downstream Fine-Tuning:

• Linear evaluation: freeze representations; train linear classifier on downstream task
• Full fine-tuning: also update representation parameters on downstream task; slight improvements
• Transfer quality: downstream accuracy reflects representation quality; benchmark for unsupervised method quality
• Task diversity: tested on classification, detection, segmentation; strong across diverse tasks

Positive Pair Construction:

• Image augmentation: random crops, color distortion, Gaussian blur; preserve semantic content
• Augmentation strength: stronger augmentation → harder learning problem but better learned features
• Domain-specific augmentation: video contrastive (temporal consistency), 3D point clouds (rotation-invariance)
• Negative pair sampling: importance sampling (hard negatives) vs uniform sampling (standard)

Contrastive Learning Theory:

• Mutual information lower bound: contrastive loss lower bounds mutual information between views
• Optimal augmentation: theoretically optimal augmentation level balances view similarity and information content
• Connection to noise-contrastive estimation: contrastive learning related to NCE; unnormalized probability approximation

Scaling to Billion-Parameter Models:

• Foundation models: CLIP, ALIGN, LiT combine contrastive learning with language models
• Vision-language pretraining: contrastive learning between images and text descriptions
• Scale benefits: larger models, larger batches, more data → substantial improvements
• Emergent capabilities: scaling contrastive pretraining enables impressive zero-shot performance

Contrastive self-supervised learning leverages augmentation-based positive/negative pair learning — achieving competitive representations without labeled data through principles of information maximization between augmented views.