transfer learning basics

**Transfer Learning** — leveraging knowledge from a model trained on a large dataset to improve performance on a different (usually smaller) target task. **Paradigm** 1. **Pretrain**: Train a large model on massive data (ImageNet, Common Crawl, etc.) 2. **Transfer**: Use pretrained weights as initialization for your task 3. **Fine-tune**: Train on your target data with a small learning rate **Strategies** - **Feature Extraction**: Freeze pretrained layers, only train new head. Best when target data is small and similar to pretraining data - **Full Fine-tuning**: Update all layers. Best when target data is large or different from pretraining - **Layer Freezing**: Gradually unfreeze layers from top to bottom during training **Why It Works** - Early layers learn universal features (edges, textures, syntax) - These transfer across tasks - Only task-specific features need to be learned from scratch **Examples** - Vision: ImageNet pretrained ResNet/ViT → medical imaging, satellite imagery - NLP: BERT/GPT pretrained → sentiment analysis, QA, summarization **Transfer learning** is the default approach — training from scratch is rarely justified unless you have massive domain-specific datasets.