discriminative fine-tuning

**Discriminative Fine-Tuning** is the **combined strategy of using different learning rates for different layers (layer-wise LR) during fine-tuning** — a term coined by the ULMFiT paper to describe the practice of discriminating between layers based on their depth when setting hyperparameters. **What Is Discriminative Fine-Tuning?** - **Core Idea**: Each layer group gets a different learning rate: $eta_l = eta_{base} cdot gamma^{(L-l)}$ where $l$ is the layer index and $gamma$ is the decay factor. - **Motivation**: Different layers encode different levels of abstraction and require different amounts of adaptation. - **ULMFiT**: Introduced as part of the ULMFiT framework (Howard & Ruder, 2018) alongside progressive unfreezing and slanted triangular LR. **Why It Matters** - **Transfer Learning Standard**: Now a standard practice in NLP and increasingly in vision fine-tuning. - **Robust**: Reduces sensitivity to the overall learning rate choice by using relative scaling between layers. - **Synergy**: Works best when combined with progressive unfreezing and warm-up scheduling. **Discriminative Fine-Tuning** is **personalized training for each layer** — acknowledging that not all layers need the same amount of adaptation for a new task.