Knowledge Distillation is the model compression technique where a smaller student network is trained to replicate the behavior of a larger teacher network — learning not just from hard labels but from the teacher's soft probability distributions (dark knowledge) that encode inter-class similarities and decision boundaries, producing compressed models that retain 90-99% of the teacher's performance at a fraction of the size and compute.

Hinton's Key Insight

A trained classifier's output logits contain far more information than the one-hot ground truth labels. When a digit classifier predicts "7" with 90% confidence, the remaining 10% distributed over "1" (5%), "9" (3%), "2" (1%), etc. encodes structural knowledge about digit similarity. Training a student to match this full distribution transfers this relational knowledge — hence "dark knowledge."

Standard Distillation Loss

L = α · L_CE(student_logits, hard_labels) + (1-α) · T² · KL(softmax(teacher_logits/T) || softmax(student_logits/T))

• Temperature T: Softens the probability distributions, amplifying differences among non-dominant classes. T=1 is standard softmax; T=3-20 reveals more dark knowledge. The T² factor compensates for the reduced gradient magnitude at high temperatures.
• α: Balances the hard label loss (ensures correctness) with the distillation loss (transfers teacher knowledge). Typically α=0.1-0.5.

Distillation Variants

• Logit Distillation: Student matches the teacher's output logits or probabilities. The original and simplest approach.
• Feature Distillation (FitNets): Student matches intermediate feature maps (hidden layer activations) of the teacher. Requires adaptor layers to align different layer dimensions. Transfers richer structural knowledge.
• Attention Distillation: Student matches the teacher's attention maps (in transformers), learning which tokens the teacher attends to.
• Self-Distillation: The model distills itself — earlier layers learn from later layers, or the model from the previous training epoch serves as the teacher. Improves performance without a separate teacher.

Applications in LLMs

• Distilled Language Models: DistilBERT (6-layer from 12-layer BERT) retains 97% of BERT's performance at 60% size and 60% faster. DistilGPT-2 similarly compresses GPT-2.
• Proprietary-to-Open Distillation: Large proprietary models (GPT-4) generate training data that open-source models learn from — a form of implicit distillation. Alpaca, Vicuna, and many open models used this approach.
• On-Policy Distillation: The student generates its own outputs, which the teacher scores, creating a feedback loop that matches the student's own distribution rather than the teacher's decode paths.

Knowledge Distillation is the transfer learning paradigm that compresses the intelligence of large models into small ones — making state-of-the-art AI capabilities accessible on devices and at scales where the original models cannot run.