progressive unfreezing

**Progressive Unfreezing** is a **fine-tuning strategy where layers are gradually unfrozen from top to bottom during training** — starting by training only the classifier head, then progressively unfreezing deeper layers, allowing each layer to adapt without catastrophically disrupting the pre-trained features. **How Does Progressive Unfreezing Work?** - **Phase 1**: Train only the classification head (all layers frozen). - **Phase 2**: Unfreeze the last block/layer. Train with small learning rate. - **Phase 3**: Unfreeze the next deeper block. Continue training. - **Phase N**: Eventually all layers are unfrozen, training end-to-end with very small learning rate for deep layers. **Why It Matters** - **Catastrophic Forgetting Prevention**: Gradually exposing pre-trained layers to gradients prevents sudden destruction of learned features. - **Small Datasets**: Especially beneficial when downstream data is limited — avoids overfitting early layers. - **ULMFiT**: Howard & Ruder (2018) demonstrated this technique for NLP transfer learning. **Progressive Unfreezing** is **gentle adaptation** — slowly waking up each layer of the network to let it adjust to the new task without forgetting what it already knows.