similarity-preserving distillation

**Similarity-Preserving Distillation** is a **knowledge distillation method that trains the student to produce the same pairwise similarity matrix as the teacher** — ensuring that if two inputs are similar according to the teacher, they remain similar according to the student. **How Does It Work?** - **Similarity Matrix**: For a batch of N inputs, compute the N×N similarity matrix $S_{ij} = f_i^T f_j / (||f_i|| cdot ||f_j||)$. - **Loss**: Minimize the difference between teacher's and student's similarity matrices: $||S^T - S^S||_F^2$. - **Batch-Level**: Operates on the full batch similarity structure, not individual samples. **Why It Matters** - **Manifold Preservation**: Ensures the student's feature space preserves the same neighborhood structure as the teacher. - **Architecture Agnostic**: Works regardless of dimension mismatch between teacher and student (similarity is always N×N). - **Complementary**: Can be combined with standard KD loss for improved performance. **Similarity-Preserving Distillation** is **transferring the social network of features** — teaching the student which inputs should be friends (similar) and which should be strangers (dissimilar).