feature matching distillation

**Feature Matching Distillation** (FitNets) is a **knowledge distillation approach where the student is trained to match the teacher's intermediate feature representations** — not just the final output, providing deeper knowledge transfer from the teacher's internal representations. **How Does Feature Matching Work?** - **Hint Layers**: Select intermediate layers from teacher and student. - **Projection**: If dimensions differ, use a learnable linear projection ($W_s cdot F_{student} approx F_{teacher}$). - **Loss**: L2 distance between projected student features and teacher features at matched layers. - **Paper**: Romero et al., "FitNets: Hints for Thin Deep Nets" (2015). **Why It Matters** - **Deeper Transfer**: Transfers knowledge from internal representations, not just output predictions. - **Thin & Deep**: Enables training very deep, thin student networks that would otherwise be difficult to train. - **Layer Matching**: The choice of which teacher and student layers to match significantly impacts performance. **Feature Matching Distillation** is **transferring the teacher's internal thought process** — teaching the student to think like the teacher at every level, not just arrive at the same answer.