Self-Supervised Learning Pretext Tasks Representation Learning
# Self-Supervised Learning: Pretext Tasks & Representation Learning
## Introduction & Motivation
Self-supervised learning: learn representations from unlabeled data via pretext tasks. Rotation prediction: rotate image, predict angle. Colorization: predict colors from grayscale. Jigsaw puzzles: arrange shuffled patches. Contrastive methods: positive/negative pair learning. Applications: pretrain on unlabeled data, reduce supervised data needs, improve transfer.
Motivation: Labeled data expensive, unlabeled abundant. Self-supervision via pretext tasks learns general features.
Applications: Pre-training, limited labels, cross-domain transfer.
---
## Core Concepts & Theory
### Pretext Task
Auxiliary task without labels; learning signal from data itself.
### Contrastive Learning
Maximize similarity of related samples, minimize unrelated.
### Clustering-based Methods
Group similar samples; learn cluster assignments.
---
## Mathematical Formulation
Contrastive (NT-Xent):
$$L = -\log \frac{e^{ ext{sim}(z_i, z_j)/ au}}{\sum_k e^{ ext{sim}(z_i, z_k)/ au}}$$
Rotation prediction loss:
$$L = ext{CrossEntropy}(p_{ ext{rotation}}, y_{ ext{rotation}})$$
Colorization loss:
$$L = ext{CrossEntropy}(p_{ ext{color}}, y_{ ext{color}})$$
---
## Advanced Theory & Extensions
### Multi-View Learning
Multiple views of same instance; leverage consistency.
### Momentum Contrast (MoCo)
Maintain queue of negative samples; efficient contrastive.
### BYOL (Bootstrap Your Own Latent)
No negative pairs; learn via exponential moving average.
---
## Computational Considerations
Pretext task: O(forward + backward) per sample.
Contrastive: O(N·forward) pairwise distances.
Clustering: O(N·K) for K clusters.
---
## Practical Implementation Strategies
### Task Selection
Choose task capturing domain structure; not too easy/hard.
### Augmentation
Strong augmentation; preserve semantic content.
### Momentum Encoder
Slowly updated encoder; stabilize learning.
---
## Benchmark Datasets & Evaluation
ImageNet: Standard pretraining; linear eval protocol.
CIFAR-10: Self-supervised baseline; ~95% accuracy.
Unlabeled Data: ImageNet-1M; YFCC100M; pretraining source.
---
## Key Challenges & Limitations
### Task Relevance
Pretext irrelevant to downstream; limited transfer.
### Augmentation Dependency
Strong augmentation critical; domain-dependent.
### Computational Cost
Contrastive methods expensive; large batch sizes.
---
## Hyperparameter Tuning
Temperature τ: 0.05-0.2; controls contrast.
Momentum α: 0.99-0.999; encoder update rate.
Batch size: 256-4096; larger improves performance.
---
## Real-World Applications & Case Studies
BERT/GPT: Masked language modeling; NLP pretraining.
Vision Transformers: ImageNet pretraining; downstream transfer.
Medical Imaging: Limited labels; SSL pretrain improves.
---
## Integration with Other Methods
SSL + Fine-Tuning → rapid downstream adaptation.
SSL + Contrastive → combine multiple views.
---
## Summary & Key Takeaways
Self-supervised learning via pretext tasks and contrastive methods learns general representations from unlabeled data, enabling efficient transfer learning.
Principles:
1. Pretext task: create learning signal without labels.
2. Contrastive: maximize intra-sample, minimize inter-sample.
3. Augmentation: critical; preserve semantic content.
4. Representation: learns general features.
5. Transfer: pretrain + downstream fine-tune.
---
---
## Appendix: Practical Labs
### Lab 1: Rotation Prediction Pretext Task
import torch
import torch.nn.functional as F
import numpy as np
def rotation_prediction_loss(images, model, angles=[0, 90, 180, 270]):
"""Pretext task: predict image rotation"""
total_loss = 0
for angle in angles:
# Rotate image
rotated = torch.rot90(images, k=angle//90)
# Predict rotation
logits = model(rotated)
angle_idx = angles.index(angle)
# Cross-entropy loss
loss = F.cross_entropy(logits, torch.full((len(images),), angle_idx).long())
total_loss += loss
return total_loss / len(angles)
# Test
np.random.seed(42)
images = torch.randn(8, 3, 32, 32)
model = torch.nn.Conv2d(3, 4, 3) # Simplified
loss = rotation_prediction_loss(images, model)
assert torch.isfinite(loss), "Loss should be finite"
assert loss >= 0, "Loss non-negative"
print("✓ Rotation prediction working")
if __name__ == "__main__":
print("Lab 1: Rotation - PASSED")### Lab 2: Colorization Pretext Task
import torch
import torch.nn.functional as F
import numpy as np
def rgb_to_gray(images):
"""Convert RGB to grayscale"""
gray = 0.299 * images[:, 0] + 0.587 * images[:, 1] + 0.114 * images[:, 2]
return gray.unsqueeze(1)
def colorization_loss(images, model):
"""Pretext task: predict color from grayscale"""
gray = rgb_to_gray(images)
# Model predicts color channels
predicted_color = model(gray)
# Loss against original colors
loss = F.mse_loss(predicted_color, images)
return loss
# Test
np.random.seed(42)
images = torch.rand(8, 3, 32, 32)
model = torch.nn.Conv2d(1, 3, 3)
loss = colorization_loss(images, model)
assert torch.isfinite(loss), "Loss finite"
assert loss >= 0, "Loss non-negative"
print("✓ Colorization working")
if __name__ == "__main__":
print("Lab 2: Colorization - PASSED")### Lab 3: Contrastive Pretext Task
import torch
import torch.nn.functional as F
import numpy as np
def contrastive_loss(z_i, z_j, temperature=0.1):
"""Contrastive loss: NT-Xent"""
batch_size = z_i.size(0)
z_i = F.normalize(z_i, dim=1)
z_j = F.normalize(z_j, dim=1)
# Similarity matrix
sim = torch.mm(z_i, z_j.T) / temperature
# Positive pairs
labels = torch.arange(batch_size)
# Loss: cross-entropy over both directions
loss_ij = F.cross_entropy(sim, labels)
loss_ji = F.cross_entropy(sim.T, labels)
return (loss_ij + loss_ji) / 2
# Test
np.random.seed(42)
z_i = torch.randn(8, 128, requires_grad=True)
z_j = torch.randn(8, 128, requires_grad=True)
loss = contrastive_loss(z_i, z_j)
assert torch.isfinite(loss), "Loss finite"
assert loss >= 0, "Loss non-negative"
print("✓ Contrastive loss working")
if __name__ == "__main__":
print("Lab 3: Contrastive - PASSED")### Lab 4: Pretext Task Evaluation
import numpy as np
def evaluate_pretext_transfer(pretrained_features, downstream_X, downstream_y):
"""Evaluate pretrained features on downstream task"""
from sklearn.linear_model import LogisticRegression
# Linear probe: train classifier on frozen features
clf = LogisticRegression(max_iter=1000)
clf.fit(pretrained_features, downstream_y)
# Accuracy
accuracy = clf.score(downstream_X, downstream_y)
return accuracy
# Test
np.random.seed(42)
pretrained_features = np.random.randn(100, 128)
downstream_X = np.random.randn(50, 128)
downstream_y = np.random.randint(0, 5, 50)
accuracy = evaluate_pretext_transfer(pretrained_features, downstream_X, downstream_y)
assert 0 <= accuracy <= 1, "Accuracy in [0,1]"
print("✓ Pretext transfer evaluation working")
if __name__ == "__main__":
print("Lab 4: Transfer - PASSED")