Self-Supervised Learning Contrastive Methods Learning from Unlabeled Data

# Self-Supervised Learning & Contrastive Methods: Learning from Unlabeled Data

## 1. Introduction & Motivation

Self-supervised learning (SSL) represents a paradigm shift: learning powerful representations from unlabeled data without explicit supervision. This addresses a fundamental limitation of supervised learning—requiring expensive labels—by leveraging data itself as supervision signal.

Contrastive learning, the dominant SSL approach, operates on a simple principle: representations should be similar for related samples and dissimilar for unrelated ones. The method requires no labels, no clustering, no generative model—only pairs of related/unrelated examples.

Motivation stems from practical necessity: labeled data expensive (medical imaging), human annotation biased (NLP), or impractical at internet scale (videos). Simultaneously, unlabeled data abundant: internet contains billions of images, videos, text. Leveraging unlabeled data crucial for scalable, unbiased models.

SSL enables:
1. Pre-training at Scale: Learn from massive unlabeled corpora, fine-tune on small labeled datasets
2. Label Efficiency: Achieve supervised performance with 10-100× fewer labels
3. Robustness: Representations learned from multiple augmentations more robust
4. Reduced Bias: Avoid learning biases encoded in annotation process

## 2. Core Concepts & Theory

### Contrastive Loss Fundamentals

Contrastive learning minimizes similarity between negative pairs (different samples) while maximizing similarity for positive pairs (augmentations of same sample).

InfoNCE loss (Information Noise-Contrastive Estimation):
$$\mathcal{L}_{ ext{contrastive}} = -\log \frac{\exp( ext{sim}(z_i, z_{i^+}) / au)}{\sum_j \exp( ext{sim}(z_i, z_j) / au)}$$

where:
- z_i, z_i+ are representations of sample and its augmentation
- z_j are other samples (negatives)
- sim(., .) is similarity (cosine), tau is temperature

Interpretation: classifies positive pair among N samples using softmax. Larger N (batch size) more challenging task, potentially learning better representations.

### Augmentation Strategy

Central to SSL: data augmentations create positive pairs. For images:
- Random crop and resize
- Color distortion (brightness, contrast, saturation, hue)
- Gaussian blur
- Random horizontal flip
- Rotation (limited angles to maintain semantics)

Strong augmentation policy crucial: weak augmentations (only flip) insufficient for representation learning. Optimal augmentation intensity dataset-dependent.

For text/speech: dropout, masking, temporal cropping, mixup preserve semantic content while forcing robustness.

### Similarity Metrics

Cosine Similarity: ext{sim}(u,v) = \frac{u \cdot v}{\|u\|\|v\|}

Normalized vectors; common in modern SSL.

Euclidean Distance: ext{dist}(u,v) = \|u - v\|_2

Less common in SSL; unnormalized vectors.

Dot Product: ext{sim}(u,v) = u \cdot v

Assumes normalized representations.

### Negative Sampling

Contrastive methods require negatives—other samples treated as "not similar." Two strategies:

In-batch Negatives: Use other samples in batch as negatives. With batch size B, each sample has B-2 in-batch negatives (excluding itself and its positive pair).

Advantage: Simple, requires single forward pass.
Disadvantage: Limited negative diversity; repeated negatives across epochs.

Momentum Buffer: Maintain queue of representations from previous iterations. Current batch samples contrasted against historical representations.

Advantage: Large, diverse negative pool (10K+ samples).
Disadvantage: Older negatives may have outdated encoder.

### Momentum Encoder

Memory-bank approach maintains moving average of encoder to smooth negative samples:
theta_m <- tau * theta_m + (1 - tau) * theta

where theta is primary encoder, theta_m momentum encoder. Slower update reduces "staleness" problem—encoder changes so fast old negatives no longer representative.

## 3. Mathematical Formulation

### InfoNCE Loss Details

Given batch of N samples with augmentations:
$$\mathcal{L} = -\frac{1}{N}\sum_{i=1}^N \log \frac{\exp( ext{sim}(z_i, z_{i^+})/ au)}{\sum_{j=1}^{2N} \exp( ext{sim}(z_i, z_j)/ au)}$$

Here 2N includes sample i and its augmentation (positive), and N-1 other samples and their augmentations (negatives).

Temperature tau controls softmax smoothness: lower tau → sharper distribution → harder negative mining; higher tau → softer distribution → easier learning but less discriminative.

### Augmentation Composition

For optimal SSL, use two independent augmentation views with distribution T:
$$ ilde{x}_i^A = \mathcal{T}(x_i), \quad ilde{x}_i^B = \mathcal{T}(x_i)$$

Key: augmentations must preserve class label semantics while removing irrelevant information.

### Normalized Embeddings

Representations typically L2 normalized before similarity:
$$\hat{z} = \frac{z}{\|z\|_2}$$

Normalizing reduces magnitude effects, focuses on direction. For cosine similarity, equals dot product of normalized vectors.

### Contrastive Learning Framework

Complete SSL pipeline:

$$ ext{Input: } x_i \xrightarrow{\mathcal{T}} ilde{x}_i^A, ilde{x}_i^B$$
$$ ilde{x}_i^A \xrightarrow{f_ heta} z_i^A \xrightarrow{ ext{norm}} \hat{z}_i^A$$
$$ ilde{x}_i^B \xrightarrow{f_ heta} z_i^B \xrightarrow{ ext{norm}} \hat{z}_i^B$$
$$\mathcal{L} = -\log \frac{\exp(\hat{z}_i^A \cdot \hat{z}_i^B / au)}{\sum_j \exp(\hat{z}_i^A \cdot \hat{z}_j^B / au)}$$

where f_theta is encoder network, z_hat^A, z_hat^B normalized representations.

## 4. Advanced Theory & Extensions

### Momentum Contrast (MoCo)

Maintains exponential moving average of encoder and queue of negative samples:

1. Encode sample with primary encoder f_q: q = f_q(x)
2. Encode positive with momentum encoder f_k: k^+ = f_k(x^+)
3. Retrieve queue (K negatives) from momentum buffer
4. Contrastive loss: \mathcal{L} = -\log \frac{\exp(q \cdot k^+ / au)}{\exp(q \cdot k^+ / au) + \sum \exp(q \cdot k^- / au)}
5. Update momentum encoder: f_k <- alpha * f_k + (1 - alpha) * f_q
6. Enqueue k^+ and dequeue oldest sample

MoCo achieves ImageNet pre-training accuracy within 1% of supervised, using only unlabeled data.

### SimCLR (Simple Contrastive Learning)

Removes momentum encoder, instead uses large batch sizes and strong augmentation:

1. Apply two random augmentations to each image: x_i -> x_tilde_i^A, x_tilde_i^B
2. Encode both: z_i^A = f(x_tilde_i^A), z_i^B = f(x_tilde_i^B)
3. Non-linear projection: z = g(h) where h is representation
4. InfoNCE loss over batch (all other augmentations are negatives)

Key insight: large batch size (4K) critical—needs diverse negatives. Projection head g(.) important: removes unnecessary information, helps generalization.

### BYOL (Bootstrap Your Own Latent)

Alternative SSL approach without negative pairs:

1. Maintain target network (EMA of online network)
2. Feed image through online network: y = f_theta(x)
3. Feed augmented image through target: y' = f_tau(x^+)
4. Loss: L = ||y - sg(y')||_2^2 where sg is stop-gradient

Key: prevent collapse (all representations identical) through:
- Target network's inertia (EMA update)
- Asymmetry between online and target networks
- Batch normalization as implicit regularizer

Surprising: BYOL works without explicit negatives, challenging traditional contrastive theory. Works well despite no theoretical justification initially found.

### Swapping Assignments Problem (SwAV)

Combines clustering and contrastive learning:

1. Compute cluster assignments: q_i = softmax(f(x_i)^T * C / tau) where C is codebook
2. Swap assignments: use cluster assignments from augmentation A as targets for augmentation B
3. Cross-entropy loss between predicted and swapped assignments

Advantages: online clustering provides implicit negatives, computational efficiency, strong performance.

### Instance Discrimination

Treat each image as separate class. Each sample has single positive (augmentation) and all others negatives.

Memory bank stores previous representations, enabling large negative sets. Simplest SSL formulation but computationally efficient.

## 5. Computational Considerations

### Batch Size Effects

Large batches critical for contrastive learning: more negatives increase task difficulty, improving learned representations.

Rule of thumb: batch size 256-4096 recommended. Smaller batches (<128) often insufficient. However, computational cost scales linearly—10K batch size requires high-end GPU clusters.

### Projection Head Design

Representations learned by main network sometimes worse than penultimate layer after removing projection head. Projection head serves:

1. Information Removal: Discards task-irrelevant information
2. Dimensional Reduction: Matches downstream task dimensionality
3. Regularization: Constraint on representation space

Common: 3-layer MLP with 2048-dim hidden. Smaller heads (128-256 dims) worse; larger (8192+ dims) overkill.

### Training Stability

Contrastive learning training can be unstable:

Loss Spikes: Periodic gradient explosion related to batch composition.
Solution: Gradient clipping, careful learning rate scheduling.

Divergence: Temperature scaling critical—too high → uniform loss, no learning; too low → vanishing gradients.

Representation Collapse: All samples map to same representation, maximizing similarity trivially. Solutions:
- Large batch size (SimCLR)
- Momentum encoder (MoCo)
- Asymmetric architecture (BYOL)
- Output normalization

### Memory Requirements

Standard contrastive learning on large datasets expensive:

GPU Memory: Batch size 4096 on ResNet50 requires ~32GB per GPU.

Storage: Large momentum buffers (10K+ samples) require additional memory.

Solutions:
- Distributed training across GPUs/TPUs
- Memory-efficient queue management
- Smaller batch sizes with data accumulation

## 6. Practical Implementation Strategies

### Pre-training Workflow

1. Dataset Preparation: Collect large unlabeled dataset (ImageNet, BigCLR for vision; Common Crawl, C4 for text)
2. Augmentation Policy: Define strong augmentation pipeline; use RandAugment or AutoAugment for optimal policy
3. Encoder Selection: Choose architecture (ResNet, Vision Transformer); must match downstream task
4. Contrastive Training: Optimize for 100-1000 epochs; learning rate decay crucial
5. Fine-tuning: Replace classification head, train on labeled data (100-1000 labels)

### Augmentation Policy Selection

Vision:
- RandAugment: Randomly select from predefined operations with random magnitude
- AutoAugment: Learned optimal policy via AutoML
- Mixup/Cutmix: Mix images or cutout regions
- Color Jitter: Brightness, contrast, saturation, hue
- Gaussian Blur: Standard deviation 0.1-2.0

Text:
- Dropout: Random word dropout
- Masking: Mask tokens and predict from context
- Permutation: Shuffle word order or sentence order
- Paraphrase: Replace with synonyms or generate similar sentences

### Transfer Learning Fine-tuning

After pre-training, fine-tune on downstream task:

1. Full Fine-tuning: Train all parameters
- Higher accuracy but risk overfitting
- Use smaller learning rates (0.1× pre-training rate)
- Early stopping on validation set

2. Feature Extraction: Freeze encoder, train only head
- Much faster training
- Good with limited labeled data (<100 samples)
- Limited adaptation to task

3. Intermediate:
- Unfreeze later layers while keeping early layers frozen
- Balance adaptation and stability

### Hyperparameter Guidelines

## 7. Benchmark Datasets & Evaluation

### Pre-training Datasets

ImageNet (Unlabeled): 1.2M images, diverse objects. Standard for vision SSL.

ImageNet-21K: 14M images, 21K classes. For large-scale SSL.

Common Crawl: Billions of web images; requires cleaning.

LAION-5B: 5.85B image-text pairs from web; largest vision-language dataset.

### Downstream Benchmarks

Linear Evaluation: Freeze encoder, train linear classifier on downstream labels. Standard metric for representation quality. ImageNet accuracy: supervised 76.5%, SimCLR 69.3%, SimCLR + fine-tuning 76.5%.

Fine-tuning: After pre-training, fine-tune on downstream task. Better accuracy but different regime (not pure representation quality).

Semi-supervised: Small labeled set (1%, 10%) + larger unlabeled. SSL pre-training enables learning from limited labels.

Transfer Learning: Test on out-of-distribution datasets (Cifar10, STL10, Places, etc.).

### Evaluation Protocols

k-NN Classification: Use learned representations directly; classify based on nearest neighbors in training set. Simple, label-free evaluation metric.

Clustering Purity: Cluster representations; measure agreement with true labels via Hungarian matching.

Downstream Task Accuracy: Standard supervised metric on target task.

## 8. Key Challenges & Limitations

### Computational Cost

Pre-training large models extremely expensive: ResNet50 on ImageNet requires 100K GPU hours at 8 V100s = ~1250 GPU-days. Democratizes only for well-resourced institutions.

### Augmentation Dependency

SSL performance heavily dependent on augmentation quality. Hand-designed policies may be suboptimal; learned policies (AutoAugment) add complexity.

Different domains need different augmentation strategies (medical images different from natural images).

### Representation Collapse

Without explicit negatives or asymmetry, representations collapse to single point. Mitigation strategies (BYOL, SwAV) reduce interpretability—unclear what prevents collapse in BYOL initially.

### Domain Gap

Pre-training on large internet data then fine-tuning on specialized domain (medical imaging, satellite images) may suffer: augmentations for natural images inappropriate for domain-specific images.

Solutions: Domain-specific augmentation policy, intermediate fine-tuning on domain-specific unlabeled data.

### Limited Semantic Understanding

Contrastive methods optimize similarity of augmentations, which focuses on low-level features. Semantic understanding (object categories, relationships) may be weak.

Example: SSL learns color features strongly; downstream tasks requiring color-invariance underperform.

## 9. Hyperparameter Tuning & Optimization

### Learning Rate Scheduling

Linear warmup followed by cosine decay standard:
lr(t) = lr_max * (t / t_warmup) if t < t_warmup, else (lr_max / 2) * (1 + cos(pi * t / T))

Warmup prevents instability early training; cosine decay avoids abrupt changes.

### Temperature Selection

Temperature τ trades off between learning difficulty and numerical stability:

• Too low (τ < 0.03): Vanishing gradients, training instability
• Too high (τ > 0.2): Uniform softmax, all samples treated equally
• Optimal (τ ≈ 0.07): Balanced learning curve

Can learn temperature as parameter via entropy regularization.

### Batch Size vs. Number of Negatives

Larger batch size (more in-batch negatives) better, but hits GPU memory limits. Trade-offs:

• Large batch (4K+): Best representations, high computational cost
• Medium batch (256-1K): Good representations, manageable cost
• Small batch + memory bank: Good representations, slightly lower quality but more efficient

### Encoder Dimension

Representation dimensionality important:

• Too small (<64): Information bottleneck, poor downstream performance
• Moderate (128-512): Good for downstream tasks, efficient
• Large (2048+): Potentially better but higher computational cost

After projection head, dimensions matter less for downstream—projection head information removal is key benefit.

## 10. Real-World Applications & Case Studies

### Medical Imaging Pre-training

MoCo pre-training on 1M unlabeled chest X-rays improves disease classification:
- Supervised only: 80% AUROC
- MoCo pre-training: 87% AUROC
- With only 1% labels: MoCo achieves 78%, supervised achieves 65%

Enables deployment in settings with limited annotation budgets.

### Video Understanding

SSL extends naturally to video: temporal consistency between frames provides "free" positive pairs.
- R3D or SlowFast encoders trained with MoCo on unlabeled video datasets
- Achieves competitive accuracy for action recognition with 10× fewer labels
- VideoClip models learn from video-text pairs; fine-tune on action recognition

### Recommendation Systems

SSL pre-training on user-item interactions before supervised recommendation:
- User/item representations from interaction graph
- Contrastive learning on different types of interactions (clicks, purchases, views)
- Improves recommendation accuracy by 10-15%, enables cold-start recommendation

### Language Models

BERT uses masked language modeling (form of self-supervision). GPT-3 uses language modeling as self-supervision.

Modern large language models (LLaMA, PaLM) pre-trained on massive unlabeled text using language modeling loss, then fine-tuned or prompted for downstream tasks.

Self-supervised pre-training enabled the language model revolution—scaling laws show 10-100× data efficiency from pre-training.

## 11. Integration with Other Methods

### Combining with Supervised Learning

Semi-supervised Learning: Use SSL for pre-training, then supervised fine-tuning. MixMatch, FixMatch combine:
- SSL via pseudo-labeling and consistency regularization
- Supervised learning on labeled subset
- Results in learning with limited labels (competitive with supervised using 10× fewer labels)

Meta-Learning: Pre-trained representations transfer well to few-shot learning. Pre-train with SSL (e.g., SimCLR), then meta-learn on downstream task. Improves few-shot accuracy by 10-20%.

### Clustering Integration

K-means on learned representations often discovers semantic clusters without labeled data.
- DeepCluster iterates: cluster representations → update labels → train supervised loss
- Combines clustering and representation learning; competitive with contrastive methods

### Knowledge Distillation

Use pre-trained teacher (large model) to distill to smaller student model:
1. Train large teacher with SSL
2. Train student by matching teacher's soft predictions
3. Results in efficient models maintaining 95%+ accuracy of teacher

Enables deployment on edge devices.

## 12. Future Research Directions

### Vision-Language Pre-training

Large-scale image-text pairs (CLIP, ALIGN) learn joint embeddings through alignment loss. Pre-training on billions of web image-text pairs enables zero-shot classification: describe object in text, classify new images without retraining.

### Temporal SSL for Video

Beyond frame consistency, use temporal order, speed, audio-visual correspondence as self-supervision signals. Temporal SSL may learn better motion/action representations than spatial SSL alone.

### Self-Supervised Outlier Detection

Use SSL representations for anomaly detection: samples far from training distribution in representation space likely anomalous.

### Prompt Learning with SSL

Pre-trained models + prompt learning (in-context learning for LLMs) enable few-shot adaptation without gradient updates. Combine SSL pre-training with prompting for flexible model adaptation.

## 13. Summary & Key Takeaways

Self-supervised learning via contrastive methods revolutionized representation learning, enabling powerful models from unlabeled data. Key insights:

1. Contrastive Learning Core: Maximize similarity between augmentations, minimize for different samples. Simple principle, powerful results.

2. Augmentation Critical: Quality of augmentation policy determines learned representations. Domain-specific strategies optimal.

3. Batch Size Matters: Larger batches (2K+) significantly improve representation quality through more diverse negatives.

4. Temperature Tuning: Balance between learning difficulty and numerical stability; small changes (0.05-0.1) significantly affect results.

5. Momentum Importance: Momentum encoders (MoCo) or asymmetric architectures (BYOL) prevent representation collapse and reduce negative staleness.

6. Transfer Learning Strength: Pre-trained SSL models transfer excellently to downstream tasks, often matching or exceeding supervised pre-training.

7. No Labels Needed: Can scale to billions of images without labels; when labels limited, pre-trained models need 10-100× fewer.

8. Computational Trade-off: Benefits from large-scale computation; requires careful optimization for efficiency.

---

## Appendix: Practical Labs

### Lab 1: SimCLR Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import models, transforms
from torch.utils.data import DataLoader, TensorDataset
import numpy as np

class SimCLR(nn.Module):
 def __init__(self, base_encoder, projection_dim=128):
 super().__init__()
 self.encoder = base_encoder
 # Remove classification head
 self.encoder.fc = nn.Identity()
 
 # Projection head: MLP with hidden layer
 dim_mlp = self.encoder.fc.in_features if hasattr(self.encoder, 'fc') else 2048
 self.projection = nn.Sequential(
 nn.Linear(dim_mlp, dim_mlp),
 nn.ReLU(),
 nn.Linear(dim_mlp, projection_dim)
 )
 
 def forward(self, x):
 h = self.encoder(x)
 z = self.projection(h)
 return F.normalize(z, dim=1)

class ContrastiveLoss(nn.Module):
 def __init__(self, temperature=0.07):
 super().__init__()
 self.temperature = temperature
 
 def forward(self, z_i, z_j, batch_size=256):
 """Compute InfoNCE loss for batch of size batch_size"""
 # Concatenate representations
 representations = torch.cat([z_i, z_j], dim=0)
 
 # Similarity matrix
 similarity_matrix = torch.mm(representations, representations.t())
 
 # Mask to remove diagonal (same sample)
 mask = torch.eye(2*batch_size, dtype=torch.bool, device=z_i.device)
 similarity_matrix.masked_fill_(mask, -9e15)
 
 # Positive pairs are (i, batch_size+i) and (batch_size+i, i)
 pos_mask = torch.zeros(2*batch_size, 2*batch_size, dtype=torch.bool, device=z_i.device)
 for i in range(batch_size):
 pos_mask[i, batch_size + i] = True
 pos_mask[batch_size + i, i] = True
 
 # Contrastive loss
 logits = similarity_matrix / self.temperature
 loss = 0
 for i in range(2*batch_size):
 pos = logits[i][pos_mask[i]].sum()
 neg = torch.logsumexp(logits[i], dim=0)
 loss += -pos + neg
 
 return loss / (2*batch_size)

# Augmentation pipeline
aug_train = transforms.Compose([
 transforms.RandomCrop(32, padding=4),
 transforms.RandomHorizontalFlip(),
 transforms.ColorJitter(brightness=0.4, contrast=0.4, saturation=0.4, hue=0.1),
 transforms.GaussianBlur(kernel_size=3),
 transforms.ToTensor(),
 transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
])

# Training loop (simplified)
device = 'cuda' if torch.cuda.is_available() else 'cpu'
model = SimCLR(models.resnet18(pretrained=False), projection_dim=128).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = ContrastiveLoss(temperature=0.07)

print("SimCLR model initialized successfully")

### Lab 2: MoCo with Memory Bank

import torch
import torch.nn as nn
import torch.nn.functional as F
from collections import deque

class MoCo(nn.Module):
 def __init__(self, base_encoder, dim=128, queue_size=65536, momentum=0.999):
 super().__init__()
 self.momentum = momentum
 self.queue_size = queue_size
 
 # Encoder
 self.encoder_q = base_encoder
 self.encoder_k = base_encoder # Copy for momentum encoder
 self._init_momentum_encoder()
 
 # Memory bank (queue)
 self.register_buffer("queue", torch.randn(dim, queue_size))
 self.register_buffer("queue_ptr", torch.zeros(1, dtype=torch.long))
 
 def _init_momentum_encoder(self):
 """Copy encoder_q to encoder_k"""
 for param_q, param_k in zip(self.encoder_q.parameters(), 
 self.encoder_k.parameters()):
 param_k.data.copy_(param_q.data)
 param_k.requires_grad = False
 
 @torch.no_grad()
 def _update_momentum_encoder(self):
 """Update momentum encoder with exponential moving average"""
 for param_q, param_k in zip(self.encoder_q.parameters(),
 self.encoder_k.parameters()):
 param_k.data = param_k.data * self.momentum + param_q.data * (1 - self.momentum)
 
 @torch.no_grad()
 def _dequeue_and_enqueue(self, keys):
 """Update queue with new keys"""
 batch_size = keys.shape[0]
 ptr = int(self.queue_ptr)
 
 # Replace oldest keys
 if ptr + batch_size <= self.queue_size:
 self.queue[:, ptr:ptr + batch_size] = keys.t()
 else:
 # Wrap around
 self.queue[:, ptr:] = keys[:self.queue_size - ptr].t()
 self.queue[:, :batch_size - (self.queue_size - ptr)] = keys[self.queue_size - ptr:].t()
 
 ptr = (ptr + batch_size) % self.queue_size
 self.queue_ptr[0] = ptr
 
 def forward(self, im_q, im_k):
 """Return logits and targets"""
 # Encode query
 q = self.encoder_q(im_q)
 q = F.normalize(q, dim=1)
 
 with torch.no_grad():
 self._update_momentum_encoder()
 k = self.encoder_k(im_k)
 k = F.normalize(k, dim=1)
 
 # Logits: (B, 1+K)
 l_pos = torch.einsum('nc,nc->n', [q, k]).unsqueeze(-1) # (B, 1)
 l_neg = torch.einsum('nc,ck->nk', [q, self.queue.clone().detach()]) # (B, K)
 logits = torch.cat([l_pos, l_neg], dim=1)
 
 # Update queue
 self._dequeue_and_enqueue(k)
 
 return logits

# Loss with temperature
def moco_loss(logits, temperature=0.07):
 labels = torch.zeros(logits.shape[0], dtype=torch.long, device=logits.device)
 return F.cross_entropy(logits / temperature, labels)

print("MoCo model initialized successfully")

### Lab 3: BYOL Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F
from copy import deepcopy

class BYOL(nn.Module):
 def __init__(self, base_encoder, projection_dim=256, hidden_dim=4096):
 super().__init__()
 
 # Online network
 self.online_network = nn.Sequential(
 base_encoder,
 nn.Linear(2048, hidden_dim),
 nn.BatchNorm1d(hidden_dim),
 nn.ReLU(),
 nn.Linear(hidden_dim, projection_dim)
 )
 
 # Target network (EMA of online)
 self.target_network = deepcopy(self.online_network)
 for param in self.target_network.parameters():
 param.requires_grad = False
 
 # Predictor head (asymmetry)
 self.predictor = nn.Sequential(
 nn.Linear(projection_dim, hidden_dim),
 nn.BatchNorm1d(hidden_dim),
 nn.ReLU(),
 nn.Linear(hidden_dim, projection_dim)
 )
 
 self.momentum = 0.99
 
 @torch.no_grad()
 def update_target_network(self):
 """Update target network with exponential moving average"""
 for online_param, target_param in zip(self.online_network.parameters(),
 self.target_network.parameters()):
 target_param.data = self.momentum * target_param.data + (1 - self.momentum) * online_param.data
 
 def forward(self, x1, x2):
 # Online: encode and predict
 online_proj_1 = self.online_network(x1)
 pred_1 = self.predictor(online_proj_1)
 
 online_proj_2 = self.online_network(x2)
 pred_2 = self.predictor(online_proj_2)
 
 # Target: no gradient, no predictor
 with torch.no_grad():
 target_proj_1 = self.target_network(x1)
 target_proj_2 = self.target_network(x2)
 
 # Loss: minimize L2 distance between predictions and target projections
 loss_1 = F.mse_loss(F.normalize(pred_1, dim=1), 
 F.normalize(target_proj_2, dim=1), reduction='mean')
 loss_2 = F.mse_loss(F.normalize(pred_2, dim=1),
 F.normalize(target_proj_1, dim=1), reduction='mean')
 
 return (loss_1 + loss_2) / 2

print("BYOL model initialized successfully")

### Lab 4: Pre-training and Fine-tuning Pipeline

import torch
import torch.nn as nn
from torchvision import models, datasets, transforms
from torch.utils.data import DataLoader

# Pre-training on full dataset
def pretrain():
 device = 'cuda' if torch.cuda.is_available() else 'cpu'
 
 # Load encoder
 encoder = models.resnet18(pretrained=False)
 
 # Build contrastive model
 model = SimCLR(encoder, projection_dim=128).to(device)
 optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
 
 # Dummy data
 X = torch.randn(1000, 3, 32, 32)
 dataset = TensorDataset(X)
 loader = DataLoader(dataset, batch_size=64, shuffle=True)
 
 # Pre-training (simplified: 5 epochs)
 for epoch in range(5):
 for batch in loader:
 x = batch[0].to(device)
 # Two augmentations of same batch
 x1, x2 = x, x + 0.1*torch.randn_like(x)
 z1 = model(x1)
 z2 = model(x2)
 
 # Dummy contrastive loss (simplified)
 loss = -torch.mean(F.cosine_similarity(z1, z2))
 optimizer.zero_grad()
 loss.backward()
 optimizer.step()
 
 print(f"Pre-train Epoch {epoch+1}: Loss {loss:.4f}")
 
 return encoder

# Fine-tuning on downstream task
def finetune(encoder):
 device = 'cuda' if torch.cuda.is_available() else 'cpu'
 
 # Remove projection head
 model = nn.Sequential(*list(encoder.children())[:-1]) # Remove fc
 
 # Add classification head
 classifier = nn.Sequential(
 model,
 nn.Flatten(),
 nn.Linear(512, 10) # 10 classes
 ).to(device)
 
 optimizer = torch.optim.SGD(classifier.parameters(), lr=0.01)
 criterion = nn.CrossEntropyLoss()
 
 # Dummy downstream data
 X = torch.randn(500, 3, 32, 32)
 y = torch.randint(0, 10, (500,))
 dataset = TensorDataset(X, y)
 loader = DataLoader(dataset, batch_size=32, shuffle=True)
 
 # Fine-tune
 for epoch in range(3):
 for x, labels in loader:
 x, labels = x.to(device), labels.to(device)
 logits = classifier(x)
 loss = criterion(logits, labels)
 optimizer.zero_grad()
 loss.backward()
 optimizer.step()
 
 acc = torch.argmax(classifier(X.to(device)), dim=1).eq(y.to(device)).float().mean()
 print(f"Fine-tune Epoch {epoch+1}: Loss {loss:.4f}, Acc {acc:.4f}")

# encoder = pretrain()
# finetune(encoder)
print("Pre-training and fine-tuning pipeline ready")