model distillation knowledge

**Knowledge Distillation** is the **model compression technique where a smaller student network is trained to replicate the behavior of a larger teacher network — learning not just from hard labels but from the teacher's soft probability distributions (dark knowledge) that encode inter-class similarities and decision boundaries, producing compressed models that retain 90-99% of the teacher's performance at a fraction of the size and compute**. **Hinton's Key Insight** A trained classifier's output logits contain far more information than the one-hot ground truth labels. When a digit classifier predicts "7" with 90% confidence, the remaining 10% distributed over "1" (5%), "9" (3%), "2" (1%), etc. encodes structural knowledge about digit similarity. Training a student to match this full distribution transfers this relational knowledge — hence "dark knowledge." **Standard Distillation Loss** L = α · L_CE(student_logits, hard_labels) + (1-α) · T² · KL(softmax(teacher_logits/T) || softmax(student_logits/T)) - **Temperature T**: Softens the probability distributions, amplifying differences among non-dominant classes. T=1 is standard softmax; T=3-20 reveals more dark knowledge. The T² factor compensates for the reduced gradient magnitude at high temperatures. - **α**: Balances the hard label loss (ensures correctness) with the distillation loss (transfers teacher knowledge). Typically α=0.1-0.5. **Distillation Variants** - **Logit Distillation**: Student matches the teacher's output logits or probabilities. The original and simplest approach. - **Feature Distillation (FitNets)**: Student matches intermediate feature maps (hidden layer activations) of the teacher. Requires adaptor layers to align different layer dimensions. Transfers richer structural knowledge. - **Attention Distillation**: Student matches the teacher's attention maps (in transformers), learning which tokens the teacher attends to. - **Self-Distillation**: The model distills itself — earlier layers learn from later layers, or the model from the previous training epoch serves as the teacher. Improves performance without a separate teacher. **Applications in LLMs** - **Distilled Language Models**: DistilBERT (6-layer from 12-layer BERT) retains 97% of BERT's performance at 60% size and 60% faster. DistilGPT-2 similarly compresses GPT-2. - **Proprietary-to-Open Distillation**: Large proprietary models (GPT-4) generate training data that open-source models learn from — a form of implicit distillation. Alpaca, Vicuna, and many open models used this approach. - **On-Policy Distillation**: The student generates its own outputs, which the teacher scores, creating a feedback loop that matches the student's own distribution rather than the teacher's decode paths. Knowledge Distillation is **the transfer learning paradigm that compresses the intelligence of large models into small ones** — making state-of-the-art AI capabilities accessible on devices and at scales where the original models cannot run.