Transfer Learning Fine-Tuning Leveraging Pretrained Models
# Transfer Learning & Fine-Tuning: Leveraging Pretrained Models
## Introduction & Motivation
Transfer learning: leverage pretrained weights from large datasets. Fine-tuning: adapt to target task via continued training. Feature extraction: freeze backbone, train head only. Catastrophic forgetting: prevent via low learning rates, layer freezing. Applications: limited data tasks, rapid deployment, domain adaptation.
Motivation: Pretraining on large data (ImageNet, BERT) captures general features. Fine-tuning leverages this; faster convergence, better performance with limited data.
Applications: Computer vision, NLP, medical imaging.
---
## Core Concepts & Theory
### Feature Extraction
Freeze pretrained encoder; train task-specific head. Fast, memory-efficient.
### Fine-Tuning
Unfreeze and retrain encoder layers; better adaptation but slower.
### Layer Freezing Strategy
Freeze early layers (generic); fine-tune later layers (task-specific).
---
## Mathematical Formulation
Fine-tuning loss:
$$L = L_{ ext{task}}(\hat{y}, y) + \lambda \|w - w_{ ext{pretrained}}\|^2$$
regularization term prevents drift from pretrained weights.
Learning rate scheduling:
$$\alpha_{ ext{fine-tune}} = \alpha_{ ext{pretrain}} / k$$
typically k = 10-100 (lower LR to avoid disruption).
---
## Advanced Theory & Extensions
### Progressive Unfreezing
Gradually unfreeze layers from top to bottom; controlled adaptation.
### Discriminative Learning Rates
Different LR per layer; higher for later layers.
### Domain Adaptation
Minimize domain shift; adversarial or distribution matching.
---
## Computational Considerations
Feature extraction: O(forward + train_head); fast.
Fine-tuning: O(L·forward + L·backward); slower than head training.
Memory: Full model in memory; ~same as pretraining.
---
## Practical Implementation Strategies
### Layer Freezing
Freeze backbone initially; unfreeze progressively.
### Learning Rate Selection
10× lower than pretraining; or discriminative LR per layer.
### Regularization
L2 penalty toward pretrained weights; prevent drift.
---
## Benchmark Datasets & Evaluation
ImageNet Pretrained: Standard for vision; ResNet, EfficientNet.
BERT Pretrained: Standard for NLP; fine-tune for downstream tasks.
Medical Imaging: Transfer from ImageNet; improves with limited data.
---
## Key Challenges & Limitations
### Catastrophic Forgetting
High LR → relearn pretraining. Use low LR, layer freezing.
### Domain Mismatch
Pretraining domain ≠ target domain; adaptation needed.
### Negative Transfer
Wrong pretrained model; choose related pretraining task.
---
## Hyperparameter Tuning
Learning rate: 10-100× lower than pretraining; discriminative LR.
Frozen layers: Start frozen; unfreeze gradually.
Regularization: L2 penalty; prevent drift from pretrained.
---
## Real-World Applications & Case Studies
ImageNet → Medical Imaging: Common for small datasets; improves 5-10%.
BERT → Downstream NLP: Standard; fine-tune for each task.
Zero-Shot Transfer: Use pretrained, no fine-tuning; feature extraction.
---
## Integration with Other Methods
Transfer + Data Augmentation → maximize limited data.
Transfer + Ensemble → combine multiple pretrained models.
---
## Summary & Key Takeaways
Transfer learning via feature extraction and fine-tuning leverages pretrained models, achieving superior performance with limited data through careful learning rate and layer freezing strategies.
Principles:
1. Feature extraction: freeze backbone, train head; fast.
2. Fine-tuning: adapt via low LR; prevent forgetting.
3. Progressive unfreezing: gradually unfreeze layers.
4. Discriminative LR: higher for later layers.
5. Regularization: L2 penalty toward pretrained.
---
---
## Appendix: Practical Labs
### Lab 1: Feature Extraction
import torch
import torch.nn as nn
class FeatureExtractorModel(nn.Module):
def __init__(self, backbone, n_classes=10):
super().__init__()
self.backbone = backbone
# Freeze backbone
for param in self.backbone.parameters():
param.requires_grad = False
# Task-specific head
self.head = nn.Linear(512, n_classes)
def forward(self, x):
features = self.backbone(x)
output = self.head(features)
return output
# Test
backbone = nn.Sequential(nn.Linear(100, 512), nn.ReLU())
model = FeatureExtractorModel(backbone, n_classes=10)
x = torch.randn(32, 100)
output = model(x)
assert output.shape == (32, 10), "Output shape correct"
assert not model.backbone[0].weight.requires_grad, "Backbone frozen"
assert model.head.weight.requires_grad, "Head trainable"
print("✓ Feature extraction working")
if __name__ == "__main__":
print("Lab 1: Feature Extraction - PASSED")### Lab 2: Fine-Tuning with Low Learning Rate
import torch
import torch.nn as nn
import numpy as np
def finetune_with_low_lr(model, X_train, y_train, lr_pretrain=0.001):
"""Fine-tune model with 10× lower learning rate"""
lr_finetune = lr_pretrain / 10.0
optimizer = torch.optim.SGD(model.parameters(), lr=lr_finetune)
losses = []
for epoch in range(5):
loss = 0
for i in range(len(X_train)):
output = model(X_train[i:i+1])
mse_loss = ((output - y_train[i:i+1]) ** 2).mean()
optimizer.zero_grad()
mse_loss.backward()
optimizer.step()
loss += mse_loss.item()
losses.append(loss / len(X_train))
return losses
# Test
np.random.seed(42)
model = nn.Sequential(nn.Linear(100, 50), nn.ReLU(), nn.Linear(50, 10))
X_train = torch.randn(32, 100)
y_train = torch.randn(32, 10)
losses = finetune_with_low_lr(model, X_train, y_train)
assert len(losses) == 5, "Should have 5 epochs"
assert losses[-1] < losses[0], "Loss should decrease"
print("✓ Fine-tuning with low LR working")
if __name__ == "__main__":
print("Lab 2: FineTune - PASSED")### Lab 3: Layer Freezing Strategy
import torch
import torch.nn as nn
def set_layer_freeze(model, freeze_layers=None):
"""Freeze specified layers"""
for name, param in model.named_parameters():
if freeze_layers and any(layer in name for layer in freeze_layers):
param.requires_grad = False
else:
param.requires_grad = True
# Test
model = nn.Sequential(
nn.Linear(100, 64),
nn.ReLU(),
nn.Linear(64, 32),
nn.ReLU(),
nn.Linear(32, 10)
)
set_layer_freeze(model, freeze_layers=['0', '2'])
trainable_count = sum(1 for p in model.parameters() if p.requires_grad)
total_count = sum(1 for p in model.parameters())
assert trainable_count < total_count, "Some layers should be frozen"
print("✓ Layer freezing working")
if __name__ == "__main__":
print("Lab 3: Freezing - PASSED")### Lab 4: Discriminative Learning Rates
import torch
import torch.nn as nn
def get_discriminative_lr(model, base_lr=0.001, decay=0.1):
"""Set discriminative learning rates (lower for earlier layers)"""
params = []
for name, param in model.named_parameters():
# Extract layer number
if 'linear' in name or '0' in name or '1' in name or '2' in name:
layer_depth = int(name.split('.')[0])
lr = base_lr * (decay ** (2 - layer_depth))
else:
lr = base_lr
params.append({'params': param, 'lr': lr})
return params
# Test
model = nn.Sequential(
nn.Linear(100, 64),
nn.Linear(64, 32),
nn.Linear(32, 10)
)
param_groups = get_discriminative_lr(model, base_lr=0.001)
lrs = [pg['lr'] for pg in param_groups]
assert len(lrs) > 0, "Should have learning rates"
assert all(lr > 0 for lr in lrs), "All LRs positive"
print("✓ Discriminative LR working")
if __name__ == "__main__":
print("Lab 4: DiscriminativeLR - PASSED")