attention transfer

**Attention Transfer** is a **feature-based knowledge distillation method where the student is trained to mimic the teacher's spatial attention maps** — ensuring the student focuses on the same image regions as the teacher, transferring "what to look at" rather than just "what to predict." **How Does Attention Transfer Work?** - **Attention Map**: $A = sum_c |F_c|^p$ where $F_c$ is the feature map of channel $c$ and $p$ controls the power. - **Loss**: L2 distance between normalized teacher and student attention maps at each layer. - **Layers**: Attention is transferred from multiple intermediate layers simultaneously. - **Paper**: Zagoruyko & Komodakis, "Paying More Attention to Attention" (2017). **Why It Matters** - **Interpretable**: Directly transfers the spatial focus pattern from teacher to student. - **Complementary**: Can be combined with logit-based distillation for stronger knowledge transfer. - **Efficiency**: Small additional computational cost — attention maps are cheap to compute. **Attention Transfer** is **teaching the student where to look** — transferring the teacher's spatial focus patterns to guide the student's feature learning.