knowledge distillation variants

**Knowledge Distillation Variants** are **extensions of the original Hinton et al. (2015) teacher-student distillation framework** — encompassing different ways to transfer knowledge from a larger model to a smaller one, including response-based, feature-based, and relation-based approaches. **Major Variants** - **Response-Based**: Student mimics teacher's soft output probabilities (original KD). Loss: KL divergence on softened logits. - **Feature-Based** (FitNets): Student mimics teacher's intermediate feature representations. Requires projection layers for dimension matching. - **Relation-Based** (RKD): Student preserves the relational structure (distances, angles) between samples as computed by the teacher. - **Attention Transfer**: Student mimics teacher's attention maps (spatial or channel attention). **Why It Matters** - **Flexibility**: Different variants are optimal for different architectures and tasks. - **Complementary**: Multiple distillation signals can be combined for stronger compression. - **Scale**: Used to compress billion-parameter LLMs into practical deployment-sized models. **Knowledge Distillation Variants** are **the different channels of knowledge transfer** — each capturing a different aspect of what the teacher model knows.