Transfer Learning Fine-Tuning

# Transfer Learning & Fine-Tuning

## Introduction & Motivation

Transfer Learning: leverage pre-trained models. Fine-tuning, feature extraction. Applications: limited data domains, faster convergence.

Motivation: Reuse learned representations.

Applications: Medical imaging, domain adaptation, low-data scenarios.

---

## Core Concepts & Theory

### Pre-training

Train on large dataset.

### Fine-tuning

Adapt to target task.

### Feature Extraction

Use frozen backbone.

### Domain Adaptation

Minimize distribution shift.

---

## Mathematical Formulation

Transfer Loss:
$$L_{ ext{target}} = L_{ ext{task}}(f_{ ext{frozen}}(x), y) + \lambda L_{ ext{adapt}}$$

Feature Alignment:
$$L_{ ext{adapt}} = \|E_s[f(x_s)] - E_t[f(x_t)]\|^2$$

Selective Fine-tuning:
$$ heta^* = \arg\min_ heta L( ext{new data}) ext{ with } heta_{ ext{frozen}}$$

---

## Advanced Theory & Extensions

### Progressive Unfreezing

Layer-wise fine-tuning schedule.

### Knowledge Distillation

Transfer knowledge from teacher.

### Multi-Task Transfer

Shared representations.

---

## Computational Considerations

Feature extraction: O(H·W·C).

Fine-tuning: O(layers·params).

Gradient computation: O(depth·width).

---

## Practical Implementation Strategies

### Layer Freezing

Freeze early layers initially.

### Learning Rate Scheduling

Lower rates for pre-trained layers.

### Regularization

Prevent catastrophic forgetting.

---

## Benchmark Datasets & Evaluation

ImageNet Pre-training: Standard practice.

Medical Datasets: Limited size tasks.

CIFAR-10/100: Transfer validation.

---

## Key Challenges & Limitations

### Negative Transfer

Poor source-target alignment.

### Overfitting

Limited target data.

### Computational Cost

Storing multiple models.

---

## Hyperparameter Tuning

Fine-tuning LR: 1e-5 to 1e-3.

Freeze depth: 0-75% of layers.

Regularization: 0.0001-0.001.

---

## Real-World Applications & Case Studies

Medical Diagnosis: Limited labeled data.

Domain-Specific Tasks: Adaptation strategies.

Few-Shot Learning: Rapid adaptation.

---

## Integration with Other Methods

Transfer + data augmentation for robust adaptation; + multi-task learning for shared representations.

---

## Summary & Key Takeaways

Transfer Learning accelerates convergence via pre-trained representations.

Principles:
1. Pre-training: Large dataset learning.
2. Fine-tuning: Task adaptation.
3. Layer freezing: Preserve learned features.
4. Learning rates: Adjust per layer.
5. Regularization: Prevent forgetting.

---

## Appendix: Practical Labs

### Lab 1: Feature Extraction

import numpy as np

def extract_features(backbone, data, freeze=True):
 """Extract features from frozen backbone"""
 if freeze:
 pass
 
 features = backbone @ data.T
 return features

np.random.seed(42)
backbone = np.random.randn(512, 2048)
data = np.random.randn(32, 2048)
features = extract_features(backbone, data, freeze=True)
assert features.shape[0] == 512, "Correct feature dimension"
print("✓ Feature extraction working")

### Lab 2: Layer Unfreezing

import numpy as np

def progressive_unfreezing(num_layers, unfreeze_schedule):
 """Progressive unfreezing schedule"""
 unfrozen = set()
 
 for epoch, layer_id in enumerate(unfreeze_schedule):
 unfrozen.add(layer_id)
 
 frozen = set(range(num_layers)) - unfrozen
 return unfrozen, frozen

num_layers = 12
schedule = [11, 10, 9, 8]
unfrozen, frozen = progressive_unfreezing(num_layers, schedule)
assert len(unfrozen) == 4, "Correct unfrozen count"
assert len(frozen) == 8, "Correct frozen count"
print("✓ Progressive unfreezing working")

### Lab 3: Differential Learning Rate

import numpy as np

def differential_learning_rates(num_layers, base_lr, decay_factor=0.1):
 """Generate differential learning rates"""
 lrs = []
 
 for i in range(num_layers):
 lr = base_lr * (decay_factor ** (num_layers - i - 1))
 lrs.append(lr)
 
 return lrs

base_lr = 1e-3
lrs = differential_learning_rates(num_layers=12, base_lr=base_lr)
assert len(lrs) == 12, "Correct LR count"
assert lrs[-1] > lrs[0], "Increasing LR per layer"
print("✓ Differential learning rates working")

### Lab 4: Warmup Scheduler

import numpy as np

def linear_warmup(current_epoch, warmup_epochs, base_lr):
 """Linear warmup schedule"""
 if current_epoch < warmup_epochs:
 lr = base_lr * (current_epoch / warmup_epochs)
 else:
 lr = base_lr
 
 return lr

warmup_lr_1 = linear_warmup(current_epoch=2, warmup_epochs=5, base_lr=1e-3)
warmup_lr_2 = linear_warmup(current_epoch=10, warmup_epochs=5, base_lr=1e-3)
assert warmup_lr_1 < warmup_lr_2, "Increasing warmup rate"
print("✓ Warmup scheduler working")

---