Self-Supervised Learning Contrastive Representation Learning
# Self-Supervised Learning: Contrastive & Representation Learning
## Introduction & Motivation
Self-Supervised Learning: learn from unlabeled data. Contrastive Learning: maximize similarity for same samples, minimize for different. SimCLR, MoCo methods. Applications: pretraining, feature learning, downstream tasks.
Motivation: Leverage unlabeled data; reduce annotation needs.
Applications: Pretraining, unsupervised representation.
---
## Core Concepts & Theory
### Contrastive Objectives
Maximize agreement between augmentations.
### Data Augmentation
Critical for self-supervised learning.
### Negative Sampling
Distinguish from other samples.
---
## Mathematical Formulation
Contrastive loss (NT-Xent):
$$L_i = -\log \frac{\exp( ext{sim}(z_i, z_j) / au)}{\sum_{k=1}^{2N} \mathbb{1}_{[k
eq i]} \exp( ext{sim}(z_i, z_k) / au)}$$
SimCLR framework:
$$ ext{sim}(z_i, z_j) = \frac{z_i^T z_j}{\|z_i\| \|z_j\|}$$
---
## Advanced Theory & Extensions
### MoCo (Momentum Contrast)
Queue-based negative sampling.
### BYOL (Bootstrap Your Own Latent)
No negative pairs; target network.
### SwAV (Swapped Assignment with Views)
Clustering-based contrast.
---
## Computational Considerations
SimCLR: O(batch_size²) for similarity.
MoCo: O(queue_size·feature_dim) memory.
Training: Large batches; distributed.
---
## Practical Implementation Strategies
### Temperature Scaling
Control confidence; τ ≈ 0.07.
### Batch Size
Large batches critical; 256+.
### Learning Rate
Often higher than supervised; 0.3.
---
## Benchmark Datasets & Evaluation
ImageNet: Standard pretraining.
Linear Evaluation: Frozen backbone.
Transfer Learning: Fine-tuning performance.
---
## Key Challenges & Limitations
### Computational Cost
Large batches; GPU memory.
### Hyperparameter Sensitivity
Temperature, augmentation strength.
### Collapse
Trivial solutions; contrastive helps.
---
## Hyperparameter Tuning
Temperature τ: 0.05-0.1; confidence.
Augmentation strength: Domain-specific.
Batch size: 256-4096; larger better.
---
## Real-World Applications & Case Studies
Image Pretraining: ImageNet features.
Video Learning: Temporal coherence.
Multimodal Learning: Vision-text pairs.
---
## Integration with Other Methods
Self-supervised + Supervised → semi-supervised.
Self-supervised + Fine-tune → transfer.
---
## Summary & Key Takeaways
Self-Supervised Learning via contrastive methods enables powerful unsupervised pretraining through sample similarity maximization and augmentation-invariant features.
Principles:
1. Contrastive: similarity vs. dissimilarity.
2. Augmentations: critical for learning.
3. Temperature: confidence control.
4. Batch size: negative sampling.
5. Pretraining: downstream transfer.
---
---
## Appendix: Practical Labs
### Lab 1: Contrastive Loss (NT-Xent)
import numpy as np
def nt_xent_loss(z_i, z_j, temperature=0.07):
"""NT-Xent (Normalized Temperature-scaled Cross Entropy) loss"""
# Normalize
z_i_norm = z_i / (np.linalg.norm(z_i, axis=1, keepdims=True) + 1e-8)
z_j_norm = z_j / (np.linalg.norm(z_j, axis=1, keepdims=True) + 1e-8)
# Similarity matrix
similarity = z_i_norm @ z_j_norm.T / temperature
# Labels: diagonal are positive pairs
batch_size = len(z_i)
labels = np.arange(batch_size)
# Cross-entropy loss
exp_sim = np.exp(similarity - np.max(similarity, axis=1, keepdims=True))
probs = exp_sim / exp_sim.sum(axis=1, keepdims=True)
loss = -np.log(probs[np.arange(batch_size), labels] + 1e-8).mean()
return loss
# Test
np.random.seed(42)
z_i = np.random.randn(32, 128)
z_j = np.random.randn(32, 128)
loss = nt_xent_loss(z_i, z_j)
assert np.isfinite(loss), "Loss finite"
print("✓ NT-Xent loss working")
if __name__ == "__main__":
print("Lab 1: NTXentLoss - PASSED")### Lab 2: Data Augmentation Pipeline
import numpy as np
def augment_image_simple(image, color_jitter_strength=0.5, rotation_degrees=45, crop_ratio=0.8):
"""Simple augmentation pipeline for self-supervised learning"""
# Simulate augmentation (no actual image operations)
augmented = image.copy()
# Color jitter
color_noise = np.random.randn(*image.shape) * color_jitter_strength
augmented = np.clip(augmented + color_noise, -1, 1)
# Crop and resize (simulate)
h, w = image.shape[-2:]
crop_h = int(h * crop_ratio)
crop_w = int(w * crop_ratio)
# Random crop would go here; simulating with noise
crop_noise = np.random.randn(*image.shape) * 0.1
augmented = augmented + crop_noise
return augmented
# Test
np.random.seed(42)
image = np.random.randn(3, 224, 224)
aug_image = augment_image_simple(image)
assert aug_image.shape == image.shape, "Augmented shape"
print("✓ Augmentation working")
if __name__ == "__main__":
print("Lab 2: Augmentation - PASSED")### Lab 3: Momentum Update
import numpy as np
def momentum_update(theta_q, theta_k, momentum=0.999):
"""Update momentum encoder (MoCo style)"""
# Exponential moving average
theta_k_updated = momentum * theta_k + (1 - momentum) * theta_q
return theta_k_updated
# Test
np.random.seed(42)
theta_q = np.random.randn(1000) # Query network weights
theta_k = np.random.randn(1000) # Key network weights
theta_k_new = momentum_update(theta_q, theta_k, momentum=0.999)
assert theta_k_new.shape == theta_k.shape, "Shape preserved"
print("✓ Momentum update working")
if __name__ == "__main__":
print("Lab 3: MomentumUpdate - PASSED")### Lab 4: Representation Quality
import numpy as np
def evaluate_representation_quality(embeddings, labels, num_classes=10):
"""Evaluate representation quality via nearest neighbor accuracy"""
# For each sample, find nearest neighbor and check if same class
correct = 0
for i in range(len(embeddings)):
# Compute distances to all others
distances = np.linalg.norm(embeddings - embeddings[i], axis=1)
distances[i] = np.inf # Exclude self
# Nearest neighbor
nn_idx = np.argmin(distances)
# Check if same class
if labels[nn_idx] == labels[i]:
correct += 1
accuracy = correct / len(embeddings)
return accuracy
# Test
np.random.seed(42)
embeddings = np.random.randn(100, 64)
labels = np.repeat(np.arange(10), 10)
acc = evaluate_representation_quality(embeddings, labels)
assert 0 <= acc <= 1, "Accuracy in [0,1]"
print("✓ Representation quality working")
if __name__ == "__main__":
print("Lab 4: RepresentationQuality - PASSED")