Transfer Learning Fine-Tuning Adaptation
# Transfer Learning: Fine-Tuning & Adaptation
## Introduction & Motivation
Transfer Learning: leverage pretrained models. Fine-tuning on downstream tasks. Domain adaptation; source and target domains. Applications: limited data, fast training, knowledge reuse.
Motivation: Avoid training from scratch; reduce data requirements.
Applications: Computer vision, NLP, robotics.
---
## Core Concepts & Theory
### Pretraining
Train on large source dataset.
### Fine-tuning
Adapt to target task with gradient descent.
### Feature Extraction
Use frozen features from pretrained model.
---
## Mathematical Formulation
Transfer learning objective:
$$L_{ ext{target}} = L_{ ext{task}}(f(x; heta), y) + \lambda L_{ ext{reg}}( heta)$$
Fine-tuning with learning rate decay:
$$ heta_t = heta_{t-1} - \alpha_t
abla L_{ ext{target}}$$
where \alpha_t = \alpha_0 \cdot ext{decay}^t.
---
## Advanced Theory & Extensions
### Domain Adaptation
Align source and target distributions.
### Multi-task Transfer
Multiple source tasks.
### Continual Learning
Sequential task learning.
---
## Computational Considerations
Pretraining: One-time; large compute.
Fine-tuning: Fast convergence; 10-100x speedup.
Feature extraction: No training; inference only.
---
## Practical Implementation Strategies
### Learning Rate Scaling
Lower learning rate for transferred features.
### Layer Freezing
Freeze early layers; fine-tune late.
### Gradual Unfreezing
Progressively unfreeze layers.
---
## Benchmark Datasets & Evaluation
ImageNet Pretrained: Standard initialization.
BERT Pretraining: Language model transfer.
Task-specific Benchmarks: Downstream evaluation.
---
## Key Challenges & Limitations
### Domain Shift
Source and target domain mismatch.
### Catastrophic Forgetting
Lose source knowledge; overfitting.
### Negative Transfer
Target performance worse than baseline.
---
## Hyperparameter Tuning
Learning rate: 10x smaller than training from scratch.
Warmup epochs: Gradual unfreezing.
Freeze strategy: Early vs. late layers.
---
## Real-World Applications & Case Studies
Image Classification: ImageNet pretrained.
NLP: BERT, GPT fine-tuning.
Object Detection: COCO-pretrained.
---
## Integration with Other Methods
Transfer + Multi-task → diverse adaptation.
Transfer + Meta-learning → few-shot.
---
## Summary & Key Takeaways
Transfer Learning via fine-tuning enables rapid adaptation of pretrained models to target tasks with reduced data and computation.
Principles:
1. Pretraining: large-scale learning.
2. Fine-tuning: task adaptation.
3. Feature extraction: fixed representation.
4. Learning rates: differential scaling.
5. Layer freezing: selective updating.
---
---
## Appendix: Practical Labs
### Lab 1: Learning Rate Scheduling
import numpy as np
def fine_tuning_lr_schedule(base_lr, epoch, total_epochs=10, warmup_epochs=2):
"""Fine-tuning learning rate schedule"""
if epoch < warmup_epochs:
# Linear warmup
lr = base_lr * (epoch + 1) / warmup_epochs
else:
# Cosine decay
progress = (epoch - warmup_epochs) / (total_epochs - warmup_epochs)
lr = base_lr * (1 + np.cos(np.pi * progress)) / 2
return lr
# Test
base_lr = 1e-4
lrs = [fine_tuning_lr_schedule(base_lr, e, 10) for e in range(10)]
assert lrs[0] < lrs[1], "Warmup increasing"
assert lrs[-1] < lrs[len(lrs)//2], "Decay decreasing"
print("✓ LR schedule working")
if __name__ == "__main__":
print("Lab 1: LRSchedule - PASSED")### Lab 2: Layer Freezing
import numpy as np
def get_layer_groups(num_layers=12, freeze_ratio=0.5):
"""Partition layers into frozen and trainable groups"""
freeze_until = int(num_layers * freeze_ratio)
frozen_layers = list(range(0, freeze_until))
trainable_layers = list(range(freeze_until, num_layers))
return frozen_layers, trainable_layers
# Test
frozen, trainable = get_layer_groups(12, freeze_ratio=0.5)
assert len(frozen) + len(trainable) == 12, "All layers accounted"
assert len(frozen) == 6, "Correct freeze count"
print("✓ Layer freezing working")
if __name__ == "__main__":
print("Lab 2: LayerFreezing - PASSED")### Lab 3: Gradual Unfreezing
import numpy as np
def gradual_unfreezing_schedule(epoch, total_epochs=10, num_layers=12):
"""Gradually unfreeze layers"""
progress = epoch / total_epochs
unfreeze_until = int(num_layers * progress)
# Layers to unfreeze this epoch
newly_unfrozen = max(0, unfreeze_until - (epoch > 0) * 1)
return unfreeze_until
# Test
num_layers = 12
total_epochs = 10
for epoch in range(total_epochs):
unfreeze = gradual_unfreezing_schedule(epoch, total_epochs, num_layers)
assert 0 <= unfreeze <= num_layers, f"Valid unfreezing at epoch {epoch}"
print("✓ Gradual unfreezing working")
if __name__ == "__main__":
print("Lab 3: GradualUnfreezing - PASSED")### Lab 4: Domain Adaptation Loss
import numpy as np
def domain_adaptation_loss(source_logits, target_logits, y_source, alpha=0.5):
"""Compute domain adaptation loss"""
# Source classification loss
exp_src = np.exp(source_logits - np.max(source_logits, axis=1, keepdims=True))
probs_src = exp_src / exp_src.sum(axis=1, keepdims=True)
src_loss = -np.log(probs_src[np.arange(len(source_logits)), y_source] + 1e-8).mean()
# Domain discrepancy (MMD approximation)
src_mean = source_logits.mean(axis=0)
tgt_mean = target_logits.mean(axis=0)
domain_loss = np.sum((src_mean - tgt_mean) ** 2)
# Combined loss
total_loss = alpha * src_loss + (1 - alpha) * domain_loss
return total_loss
# Test
np.random.seed(42)
src_logits = np.random.randn(32, 10)
tgt_logits = np.random.randn(32, 10)
y_src = np.random.randint(0, 10, 32)
loss = domain_adaptation_loss(src_logits, tgt_logits, y_src)
assert np.isfinite(loss), "Loss finite"
print("✓ Domain adaptation working")
if __name__ == "__main__":
print("Lab 4: DomainAdaptation - PASSED")