knowledge distillation model compression

**Knowledge Distillation** is **the model compression technique where a smaller "student" network is trained to replicate the behavior of a larger, more accurate "teacher" network — learning from the teacher's soft probability outputs (which encode inter-class relationships) rather than hard ground-truth labels, achieving 90-99% of teacher accuracy at a fraction of the computational cost**. **Distillation Framework:** - **Teacher Model**: large, high-accuracy model that has been fully trained — may be an ensemble of models for even richer soft labels; teacher is frozen (not updated) during distillation - **Student Model**: compact model architecture designed for deployment — typically 3-10× fewer parameters than teacher; architecture can differ from teacher (e.g., teacher is ResNet-152, student is MobileNet) - **Temperature Scaling**: softmax outputs computed with temperature T — higher T (typically 2-20) produces softer probability distributions that reveal more information about inter-class similarities; T=1 recovers standard softmax - **Distillation Loss**: KL divergence between teacher and student soft distributions scaled by T² — combined with standard cross-entropy loss on hard labels; α parameter controls the weighting (typically α=0.5-0.9 for distillation loss) **Distillation Variants:** - **Response-Based**: student matches teacher's final output logits — simplest form; captures the teacher's class relationship knowledge encoded in soft probabilities - **Feature-Based**: student matches intermediate feature representations of the teacher — FitNets, Attention Transfer, and PKT methods align hidden layer activations, transferring structural knowledge about feature hierarchies - **Relation-Based**: student preserves the relational structure between samples as encoded by the teacher — Relational Knowledge Distillation (RKD) preserves pairwise distance and angle relationships in embedding space - **Self-Distillation**: model distills knowledge from its own deeper layers to shallower layers, or from a trained version of itself — Born-Again Networks show iterative self-distillation can progressively improve student beyond teacher accuracy **Advanced Techniques:** - **Online Distillation**: teacher and student train simultaneously, mutually learning from each other — Deep Mutual Learning shows peer networks can teach each other without a pre-trained teacher - **Data-Free Distillation**: generates synthetic training data using the teacher's batch normalization statistics or a trained generator — useful when original training data is unavailable due to privacy or storage constraints - **Task-Specific Distillation**: DistilBERT reduces BERT parameters by 40% while retaining 97% performance — uses triple loss: masked language model, distillation, and cosine embedding loss - **Multi-Teacher Distillation**: student learns from multiple teachers specializing in different domains or architectures — teacher contributions can be equally weighted or dynamically adjusted based on per-sample confidence **Knowledge distillation is the cornerstone of efficient model deployment — enabling state-of-the-art accuracy on resource-constrained devices (mobile phones, edge processors, embedded systems) by transferring the "dark knowledge" encoded in large models into compact, fast inference networks.**