Transfer Learning Domain Adaptation Knowledge Transfer

# Transfer Learning: Domain Adaptation & Knowledge Transfer

## Introduction & Motivation

Transfer learning enables leveraging knowledge from source tasks to improve performance on target tasks. The core insight: rather than learning from scratch on limited target data, initialize models from pre-trained weights learned on large-scale source data. This dramatically reduces data requirements, training time, and computational cost while improving generalization.

Motivation: Many practical problems lack large labeled datasets. Medical imaging, rare disease diagnosis, and specialized NLP tasks have few examples. Transfer learning bridges this gap via inductive bias from related, well-resourced domains.

Applications: ImageNet pre-trained vision models on domain-specific medical images; BERT on biomedical text; GPT-based fine-tuning for specific tasks. Transfer learning has become standard practice—few practitioners train deep models from scratch.

---

## Core Concepts & Theory

### Domains and Tasks

A domain is a distribution P(\mathbf{x}) over features. A task is a conditional distribution P(y | \mathbf{x}). Transfer learning addresses the gap when source domain/task differs from target.

### Transfer Paradigms

Domain Adaptation: Same task, different domain distribution. Learn mappings between domains.

Multitask Learning: Multiple related tasks in same domain. Shared representations benefit all tasks.

Fine-Tuning: Pretrain on source, minimize target loss with small learning rate.

Feature Extraction: Fix pretrained features, train only output layer on target.

---

## Mathematical Formulation

Let source task be (X_s, Y_s, P_s(y|\mathbf{x})) and target be (X_t, Y_t, P_t(y|\mathbf{x})).

Fine-tuning objective:
$$\min_{ heta} \mathcal{L}_t( heta; D_t) + \lambda \| heta - heta_s\|^2$$

where heta_s is pretrained weights, \lambda controls regularization.

Domain adaptation via maximum mean discrepancy (MMD):
$$ ext{MMD}^2 = \left\| \frac{1}{n_s} \sum_{i=1}^{n_s} \phi(\mathbf{x}_s^i) - \frac{1}{n_t} \sum_{j=1}^{n_t} \phi(\mathbf{x}_t^j) ight\|^2$$

Minimize MMD to align feature distributions.

---

## Advanced Theory & Extensions

### Domain-Adversarial Training

Learn features indistinguishable between source/target via adversarial objective:
$$\min_{ heta_f} \max_{ heta_d} \mathcal{L}_s( heta_f, heta_y) - \lambda \mathcal{L}_d( heta_f, heta_d)$$

where heta_d is domain discriminator.

### Self-Supervised Pretraining

Learn representations without labels (SimCLR, MoCo, MAE) on unlabeled source data, then fine-tune.

### Few-Shot Learning

Learn from very few target examples via meta-learning (MAML) or prototype networks.

### Continual Transfer

Sequentially learn multiple tasks without catastrophic forgetting via rehearsal, regularization, or architecture adaptation.

---

## Computational Considerations

Pretraining cost: Large-scale source data (~millions images, trillions tokens). Done once, amortized over many tasks.

Fine-tuning cost: O( ext{parameters} imes ext{target samples}). Typically fast (hours on GPU vs. days/weeks from scratch).

Memory: Can fine-tune large models via LoRA (rank-r factorization) reducing parameters by 100x.

---

## Practical Implementation Strategies

### Choosing Pretrain Data

ImageNet for vision; Wikipedia/books for NLP; domain-specific corpora for specialized tasks.

### Freezing vs. Fine-Tuning

Freeze early layers: Keep task-agnostic features, train only task-specific layers.

Fine-tune all layers: Start with small learning rate to avoid catastrophic forgetting.

Layer-wise learning rates: Lower rates for early layers, higher for later layers.

### Handling Domain Shift

Small domain gap: simple fine-tuning sufficient. Large gap: data augmentation, domain adversarial training, or self-supervised adaptation.

### Hyperparameter Tuning

Learning rate: Critical. Start lower than training from scratch (e.g., 1e-5 to 1e-4).

Batch size: May differ from pretraining. Smaller batches often work well.

Epochs: Early stopping on validation essential to avoid overfitting.

---

## Benchmark Datasets & Evaluation

Vision: ImageNet pretrain → CIFAR-10, Stanford Cars, Caltech-256 fine-tune.

NLP: BERT pretrain → GLUE, SQuAD, MRPC fine-tune.

Metrics: Target task performance (accuracy, F1, BLEU). Compare to training from scratch.

---

## Key Challenges & Limitations

### Negative Transfer

Source knowledge conflicts with target task, hurting performance. Mitigate via careful domain selection or adversarial adaptation.

### Catastrophic Forgetting

Fine-tuning overwrites source knowledge. Solutions: regularization (EWC), rehearsal (replay), architecture adaptation (adapters).

### Domain Shift

Large distribution mismatch reduces transfer benefit. Address via domain adaptation, augmentation, or selecting related source domains.

---

## Hyperparameter Tuning

Grid search: learning rate \{1e-5, 1e-4, 1e-3\}, freeze depth, warmup epochs.

---

## Real-World Applications & Case Studies

Medical Imaging: ImageNet pretrain → CT/MRI diagnosis. Reduces training data from millions to hundreds.

NLP: BERT → Domain-specific NER, sentiment, QA. Standardized pipeline.

Speech: Wav2Vec pretrain → low-resource language ASR.

---

## Integration with Other Methods

Transfer + Data Augmentation → Improved generalization.

Transfer + Ensemble → Multiple pretrained models combined.

Transfer + Active Learning → Efficiently label small target set.

---

## Future Research Directions

Continual transfer; multimodal pretraining; efficient adaptation; theoretical understanding of when transfer succeeds.

---

## Summary & Key Takeaways

Transfer learning is a cornerstone of modern ML. Pretrain on large-scale data, fine-tune on target tasks. Default approach for most practitioners.

Principles:
1. Large-scale pretraining captures general knowledge.
2. Fine-tuning on target task specializes knowledge.
3. Learning rate and regularization prevent catastrophic forgetting.
4. Domain adaptation techniques address distribution shift.
5. Few-shot and continual transfer extend applicability.

---

---

## Appendix: Practical Labs

### Lab 1: Fine-Tuning on Target Task

import torch
import torch.nn as nn
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Pretrained network (simulated)
class PretrainedNet(nn.Module):
 def __init__(self):
 super().__init__()
 self.feature_extractor = nn.Sequential(
 nn.Linear(4, 64),
 nn.ReLU(),
 nn.Linear(64, 32)
 )
 self.source_head = nn.Linear(32, 10)
 
 def forward(self, x):
 features = self.feature_extractor(x)
 return features

# Target model: freeze features, train new head
class TargetNet(nn.Module):
 def __init__(self, pretrained):
 super().__init__()
 self.feature_extractor = pretrained.feature_extractor
 self.target_head = nn.Linear(32, 3)
 
 def forward(self, x):
 features = self.feature_extractor(x)
 return self.target_head(features)

# Test on Iris
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
X_train = torch.FloatTensor(StandardScaler().fit_transform(X_train))
y_train = torch.LongTensor(y_train)

pretrained = PretrainedNet()
target_model = TargetNet(pretrained)

# Fine-tune
opt = torch.optim.Adam(target_model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

for epoch in range(20):
 opt.zero_grad()
 logits = target_model(X_train)
 loss = loss_fn(logits, y_train)
 loss.backward()
 opt.step()

print(f"Final loss: {loss.item():.4f}")
assert loss.item() < 1.0, "Training failed"
print("✓ Fine-tuning successful")

if __name__ == "__main__":
 print("Lab 1: Fine-Tuning - PASSED")

### Lab 2: Feature Extraction

import torch
import torch.nn as nn

# Pretrained model
pretrained = nn.Sequential(
 nn.Linear(10, 64),
 nn.ReLU(),
 nn.Linear(64, 32)
)

# Freeze pretrained features
for param in pretrained.parameters():
 param.requires_grad = False

# New classification head
classifier = nn.Linear(32, 5)

# Forward pass
X = torch.randn(100, 10)
features = pretrained(X)
logits = classifier(features)

print(f"Features shape: {features.shape}")
assert features.shape == (100, 32), "Shape mismatch"
print("✓ Feature extraction working")

if __name__ == "__main__":
 print("Lab 2: Feature Extraction - PASSED")

### Lab 3: Domain Adversarial Training

import torch
import torch.nn as nn

class DomainAdversarial(nn.Module):
 def __init__(self):
 super().__init__()
 self.feature_extractor = nn.Linear(20, 16)
 self.classifier = nn.Linear(16, 2)
 self.domain_discriminator = nn.Linear(16, 1)
 
 def forward(self, x, return_features=False):
 features = self.feature_extractor(x)
 class_logits = self.classifier(features)
 domain_logits = self.domain_discriminator(features)
 if return_features:
 return class_logits, domain_logits, features
 return class_logits, domain_logits

model = DomainAdversarial()

# Source and target data
X_source = torch.randn(50, 20)
y_source = torch.randint(0, 2, (50,))
X_target = torch.randn(40, 20) + 0.5 # Shifted distribution

class_loss_fn = nn.CrossEntropyLoss()
domain_loss_fn = nn.BCEWithLogitsLoss()

opt = torch.optim.Adam(model.parameters(), lr=0.01)

for epoch in range(10):
 # Train on source
 y_class, y_domain, _ = model(X_source, return_features=True)
 loss_class = class_loss_fn(y_class, y_source)
 loss_domain_s = domain_loss_fn(y_domain, torch.zeros(50, 1))
 
 # Train domain discriminator on target
 _, y_domain_t, _ = model(X_target, return_features=True)
 loss_domain_t = domain_loss_fn(y_domain_t, torch.ones(40, 1))
 
 loss = loss_class + 0.1 * (loss_domain_s + loss_domain_t)
 
 opt.zero_grad()
 loss.backward()
 opt.step()

print(f"Final loss: {loss.item():.4f}")
print("✓ Domain adversarial training working")

if __name__ == "__main__":
 print("Lab 3: Domain Adversarial - PASSED")

### Lab 4: Early Stopping

import torch
import torch.nn as nn

class SimpleNet(nn.Module):
 def __init__(self):
 super().__init__()
 self.layers = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))
 
 def forward(self, x):
 return self.layers(x)

model = SimpleNet()
opt = torch.optim.Adam(model.parameters(), lr=0.01)
loss_fn = nn.CrossEntropyLoss()

# Synthetic train/val
X_train = torch.randn(100, 10)
y_train = torch.randint(0, 2, (100,))
X_val = torch.randn(30, 10)
y_val = torch.randint(0, 2, (30,))

best_val_loss = float('inf')
patience = 3
patience_counter = 0

for epoch in range(20):
 # Train
 opt.zero_grad()
 logits = model(X_train)
 train_loss = loss_fn(logits, y_train)
 train_loss.backward()
 opt.step()
 
 # Validate
 with torch.no_grad():
 val_logits = model(X_val)
 val_loss = loss_fn(val_logits, y_val)
 
 if val_loss < best_val_loss:
 best_val_loss = val_loss
 patience_counter = 0
 else:
 patience_counter += 1
 
 if patience_counter >= patience:
 print(f"Early stopping at epoch {epoch}")
 break

print(f"Best val loss: {best_val_loss:.4f}")
assert best_val_loss < 1.0, "Validation loss too high"
print("✓ Early stopping working")

if __name__ == "__main__":
 print("Lab 4: Early Stopping - PASSED")