Contrastive Learning Simclr Self-Supervised Representation Learning
# Contrastive Learning: SimCLR & Self-Supervised Representation Learning
## Introduction & Motivation
Contrastive learning learns representations by maximizing similarity between positive pairs while minimizing similarity to negatives. SimCLR: simple framework (augmentation → projection → contrastive loss). Supervised contrastive: extends to labeled data. Self-supervised pre-training outperforms supervised on limited labels; transfer to downstream tasks. Applications: vision, NLP, multimodal (CLIP).
Motivation: Labeled data expensive. Self-supervised pre-training learns rich representations without labels. Contrastive objectives align with human perception: similar → close, dissimilar → far.
Applications: Pre-training, transfer learning, few-shot learning, multimodal understanding.
---
## Core Concepts & Theory
### Augmentation Strategy
Two views per sample via random augmentation (crop, color jitter, blur). Positive pair: same image, different augmentations.
### Contrastive Loss
NT-Xent (Normalized Temperature-scaled Cross Entropy): similarity between positive pair high, negatives low. Temperature τ controls sharpness.
### Projection Head
MLP maps encoder output to contrastive space; asymmetry between train (with head) and eval (without) critical.
---
## Mathematical Formulation
SimCLR loss (NT-Xent):
$$\ell_{i,j} = -\log \frac{\exp( ext{sim}(z_i, z_j) / au)}{\sum_{k=1}^{2N} \mathbb{I}_{[k
eq i]} \exp( ext{sim}(z_i, z_k) / au)}$$
Supervised contrastive loss:
$$\ell_i = -\frac{1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp( ext{sim}(z_i, z_p) / au)}{\sum_{a=1}^{N} \exp( ext{sim}(z_i, z_a) / au)}$$
where P(i) = positive pairs (same class).
---
## Advanced Theory & Extensions
### Momentum Contrast (MoCo)
Maintain momentum-updated queue of negatives; reduces batch size requirement.
### Siamese Networks
Twin encoders; hard negative mining; triplet loss.
### Multi-View Contrastive
More than two views per sample; cross-modal alignment (vision-language).
---
## Computational Considerations
Batch size: Large (256-4096) for sufficient negatives; distributed training required.
Augmentation overhead: O(2× forward passes) per sample.
Memory: Large negative cache (MoCo); GPU memory scales with batch size.
---
## Practical Implementation Strategies
### Temperature Tuning
τ ∈ [0.05, 0.5]; lower = sharper distinction, higher = softer.
### Augmentation Strength
Balance informativeness; too weak = easy negatives, too strong = lose semantic signal.
### Batch Normalization
Avoid feature leakage; use sync-BN across devices.
---
## Benchmark Datasets & Evaluation
ImageNet-1M: Standard pre-training; evaluate via linear probe (frozen encoder + linear layer).
CIFAR-10/100: Smaller; quick iteration.
Metrics: Linear probe top-1 accuracy, transfer learning performance downstream.
---
## Key Challenges & Limitations
### Batch Size Dependency
Small batches → weak negatives; need large distributed setup.
### Augmentation Leakage
Augmentations must not preserve class information; task-dependent.
### Computational Cost
Pre-training expensive (100-1000 GPU hours); amortized over downstream tasks.
---
## Hyperparameter Tuning
Temperature τ: 0.05-0.5; typically 0.1.
Learning rate: 0.3-0.5 (scales with batch size).
Projection hidden dimension: 2048; output dimension typically 128.
---
## Real-World Applications & Case Studies
ImageNet Pre-training: SimCLR achieves 71.3% top-1 linear probe (comparable to supervised).
Medical Imaging: Contrastive pre-training on unlabeled scans; downstream fine-tuning.
Multimodal (CLIP): Vision-language contrastive learning; zero-shot transfer.
---
## Integration with Other Methods
Contrastive + Fine-tuning → pre-train with contrastive, downstream supervised.
Contrastive + Clustering → pseudo-labels via clustering, supervised contrastive refine.
---
## Summary & Key Takeaways
Contrastive learning via SimCLR learns representations by pushing apart negative pairs and pulling together positive pairs, achieving strong self-supervised pre-training.
Principles:
1. Augmentation strategy: two views per sample, maximize positive similarity.
2. NT-Xent loss: normalized temperature-scaled cross entropy.
3. Large batch size needed for sufficient negatives.
4. Projection head asymmetry: use for training, discard for downstream.
5. Pre-training amortizes cost across downstream tasks.
---
---
## Appendix: Practical Labs
### Lab 1: NT-Xent Loss Implementation
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
def nt_xent_loss(z_i, z_j, tau=0.1):
"""NT-Xent (Normalized Temperature-scaled Cross Entropy) loss"""
batch_size = z_i.size(0)
# Normalize embeddings
z_i = F.normalize(z_i, dim=1)
z_j = F.normalize(z_j, dim=1)
# Similarity matrix (batch_size x batch_size)
sim = torch.mm(z_i, z_j.T) / tau
# Labels: positive pair along diagonal
labels = torch.arange(batch_size)
# Compute loss for both directions
loss_ij = F.cross_entropy(sim, labels)
loss_ji = F.cross_entropy(sim.T, labels)
return (loss_ij + loss_ji) / 2
# Data
z_i = torch.randn(32, 128)
z_j = torch.randn(32, 128)
loss = nt_xent_loss(z_i, z_j, tau=0.1)
print(f"NT-Xent loss: {loss:.4f}")
assert 0 < loss < 10, "Loss should be in reasonable range"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ NT-Xent loss working")
if __name__ == "__main__":
print("Lab 1: NT-Xent - PASSED")### Lab 2: Augmentation Pipeline
import torch
from torchvision import transforms
from PIL import Image
import numpy as np
def contrastive_augment(x, crop_size=224, jitter_strength=0.5):
"""Contrastive augmentation: crop, color jitter, blur"""
# Assume x is tensor [C, H, W]
# Convert to PIL for augmentation
x_np = (x.permute(1, 2, 0).cpu().numpy() * 255).astype(np.uint8)
x_pil = Image.fromarray(x_np)
augment = transforms.Compose([
transforms.RandomCrop(crop_size),
transforms.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.2, hue=0.1),
transforms.RandomHorizontalFlip(p=0.5),
transforms.ToTensor(),
])
return augment(x_pil)
# Dummy image (simulated)
x_dummy = torch.rand(3, 256, 256)
x_aug1 = contrastive_augment(x_dummy, crop_size=224)
x_aug2 = contrastive_augment(x_dummy, crop_size=224)
print(f"Augmented shape: {x_aug1.shape}")
assert x_aug1.shape == (3, 224, 224), "Should have correct shape"
assert not torch.allclose(x_aug1, x_aug2), "Augmentations should differ"
print("✓ Augmentation pipeline working")
if __name__ == "__main__":
print("Lab 2: Augmentation - PASSED")### Lab 3: Projection Head
import torch
import torch.nn as nn
class ProjectionHead(nn.Module):
def __init__(self, in_dim, hidden_dim=2048, out_dim=128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(in_dim, hidden_dim),
nn.BatchNorm1d(hidden_dim),
nn.ReLU(),
nn.Linear(hidden_dim, out_dim)
)
def forward(self, x):
return self.net(x)
# Test
encoder_dim = 512
proj_head = ProjectionHead(in_dim=encoder_dim, hidden_dim=2048, out_dim=128)
x_encoded = torch.randn(32, encoder_dim)
z = proj_head(x_encoded)
print(f"Projection output shape: {z.shape}")
assert z.shape == (32, 128), "Should have correct output dim"
assert not torch.allclose(z, torch.zeros_like(z)), "Output should be non-trivial"
print("✓ Projection head working")
if __name__ == "__main__":
print("Lab 3: Projection - PASSED")### Lab 4: Contrastive Similarity
import torch
import torch.nn.functional as F
import numpy as np
def compute_similarity_matrix(z1, z2):
"""Compute similarity matrix between two batches"""
# Normalize
z1_norm = F.normalize(z1, dim=1)
z2_norm = F.normalize(z2, dim=1)
# Cosine similarity
sim = torch.mm(z1_norm, z2_norm.T)
return sim
def extract_positive_negatives(sim, batch_size):
"""Extract positive and negative similarities"""
positive_sim = torch.diag(sim).mean()
# Negatives: off-diagonal
mask = ~torch.eye(batch_size, dtype=torch.bool)
negative_sim = sim[mask].mean()
return positive_sim, negative_sim
# Test
z_i = torch.randn(32, 128)
z_j = torch.randn(32, 128)
sim = compute_similarity_matrix(z_i, z_j)
pos_sim, neg_sim = extract_positive_negatives(sim, 32)
print(f"Positive similarity: {pos_sim:.4f}, Negative: {neg_sim:.4f}")
assert -1 <= pos_sim <= 1, "Similarity should be in [-1, 1]"
assert -1 <= neg_sim <= 1, "Similarity should be in [-1, 1]"
assert sim.shape == (32, 32), "Should have correct shape"
print("✓ Similarity computation working")
if __name__ == "__main__":
print("Lab 4: Similarity - PASSED")