transfer learning
**Transfer Learning and Domain Adaptation** — Transfer learning leverages knowledge from pre-trained models to accelerate learning on new tasks, while domain adaptation specifically addresses distribution shifts between source and target domains.
**Transfer Learning Paradigms** — Feature extraction freezes pre-trained layers and trains only new task-specific heads, preserving learned representations. Full fine-tuning updates all parameters with a small learning rate, adapting the entire network. Progressive unfreezing gradually thaws layers from top to bottom, allowing careful adaptation without catastrophic forgetting. The choice depends on dataset size, domain similarity, and computational budget.
**Fine-Tuning Best Practices** — Discriminative learning rates assign smaller rates to lower layers and larger rates to upper layers, reflecting the observation that early features are more general. Gradual unfreezing combined with discriminative rates prevents destroying useful pre-trained features. Weight initialization from pre-trained checkpoints provides dramatically better starting points than random initialization, especially for small target datasets where training from scratch would severely overfit.
**Domain Adaptation Methods** — Unsupervised domain adaptation aligns source and target feature distributions without target labels. Domain adversarial neural networks use gradient reversal layers to learn domain-invariant features. Maximum mean discrepancy minimizes distribution distance in reproducing kernel Hilbert spaces. Self-training generates pseudo-labels on target data, iteratively refining predictions through confident example selection.
**Modern Transfer Approaches** — Foundation models like CLIP, DINO, and large language models provide universal feature extractors that transfer across diverse tasks. Prompt tuning and adapter modules insert small trainable components into frozen models, achieving parameter-efficient transfer. Low-rank adaptation (LoRA) decomposes weight updates into low-rank matrices, enabling fine-tuning with minimal additional parameters while preserving the pre-trained model's knowledge.
**Transfer learning has fundamentally transformed deep learning practice, making state-of-the-art performance accessible even with limited data and compute by standing on the shoulders of massive pre-training investments.**