layer-wise learning rates

**Layer-Wise Learning Rates** is a **fine-tuning technique where different learning rates are applied to different layers of a pre-trained network** — typically using lower rates for earlier (more general) layers and higher rates for later (more task-specific) layers. **How Does It Work?** - **Decay Schedule**: LR decreases exponentially from the top layer to the bottom. E.g., if top layer LR = $10^{-3}$, each layer below uses LR × decay factor (e.g., 0.95). - **Intuition**: Early layers learn general features (edges, textures) that should change little. Later layers learn task-specific features that need more adaptation. - **Implementation**: Assign separate parameter groups with different learning rates in the optimizer. **Why It Matters** - **Better Fine-Tuning**: Consistently outperforms uniform learning rate across all layers. - **Feature Preservation**: Protects valuable low-level features from being overwritten during fine-tuning. - **Combined**: Often used with progressive unfreezing for maximum transfer learning performance. **Layer-Wise Learning Rates** are **the gradient speed limits for neural layers** — letting each level adapt at its own pace based on how much it needs to change.