self supervised learning visual

**Self-Supervised Visual Learning** is the **training paradigm that learns powerful visual representations from unlabeled images by solving pretext tasks (predicting masked patches, matching augmented views, reconstructing corrupted inputs) — eliminating the need for expensive human annotations while producing general-purpose features that transfer to downstream tasks (classification, detection, segmentation) with quality approaching or exceeding supervised ImageNet pretraining, fundamentally changing the economics of computer vision by leveraging billions of unlabeled images**. **Why Self-Supervised Learning** Labeled datasets (ImageNet: 1.2M images × 1000 classes) are expensive and limited. The internet contains billions of unlabeled images. Self-supervised learning (SSL) designs training objectives that extract supervision from the data itself — the structure of images provides the learning signal. **Contrastive Learning** **Core Idea**: Pull together representations of augmented views of the same image (positive pairs), push apart representations of different images (negative pairs). - **SimCLR**: Two random augmentations of the same image → encoder → projection head → contrastive loss (NT-Xent). Requires large batch sizes (4096-8192) for sufficient negative examples. Simple but effective. - **MoCo (Momentum Contrast)**: Maintains a large queue of negative examples (65,536) using a momentum-updated encoder — decouples batch size from negative count. MoCo v3 applies to Vision Transformers with excellent results. - **BYOL (Bootstrap Your Own Latent)**: No negative pairs! Uses a momentum-updated target network. Online network predicts target network's representation of a different augmentation. Prevents collapse via the momentum update asymmetry. **Masked Image Modeling** **Core Idea**: Mask random patches of an image, train the model to reconstruct the masked content (analogous to BERT's masked language modeling). - **MAE (Masked Autoencoder)**: Mask 75% of image patches. ViT encoder processes only the visible 25% patches (efficient). Lightweight decoder reconstructs pixel values of masked patches. Pre-training is fast (visible patches are only 25% of total) and learns excellent representations. - **BEiT**: Tokenizes image patches using a discrete VAE (dVAE). Masked patch prediction targets are discrete tokens rather than raw pixels — provides a higher-level learning target. - **I-JEPA**: Predicts representations (not pixels) of masked regions from visible context. Avoids pixel-level reconstruction bias toward texture over semantics. **Self-Distillation** - **DINO / DINOv2**: Self-distillation with no labels. Student and teacher networks (both ViTs) see different augmented views. Student is trained to match teacher's output distribution. Teacher is an exponential moving average of student. DINO produces features with remarkable emergent properties — attention maps automatically segment objects without any segmentation training. - **DINOv2**: Scaled to 142M images (LVD-142M curated dataset). The resulting ViT-g model produces general-purpose visual features that outperform supervised features on 12 benchmarks with frozen features (no fine-tuning). **Transfer Performance** | Method | ImageNet Linear Probe | Detection (COCO) | |--------|---------------------|-------------------| | Supervised ViT-B | 82.3% | 50.3 AP | | MAE ViT-B | 83.6% | 51.6 AP | | DINOv2 ViT-g | 86.5% | 55.2 AP | Self-Supervised Visual Learning is **the paradigm shift that decoupled visual representation learning from human labeling** — demonstrating that the visual world contains enough structure to teach itself, producing foundation models whose features generalize across tasks with minimal or no task-specific supervision.