Knowledge Distillation is the model compression technique where a smaller "student" network is trained to mimic the output behavior of a larger, more accurate "teacher" network — transferring the teacher's learned knowledge through soft probability distributions rather than hard labels, enabling deployment of compact models that retain 90-99% of the teacher's accuracy at a fraction of the size and computation.

Core Idea (Hinton et al., 2015)

• Teacher output (softmax with temperature T): $p_i^T = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}$.
• At high temperature (T=4-20): Softmax outputs reveal inter-class relationships (e.g., "3" looks more like "8" than like "7").
• These soft labels carry richer information than one-hot hard labels.
• Student learns to match teacher's soft distribution → learns the teacher's reasoning patterns.

Distillation Loss

$L = \alpha \cdot T^2 \cdot KL(p^T_{teacher} || p^T_{student}) + (1-\alpha) \cdot CE(y, p_{student})$

• First term: Match teacher's soft predictions (KL divergence).
• Second term: Match ground truth labels (cross-entropy).
• α: Balance between teacher guidance and ground truth (typically 0.5-0.9).
• T²: Compensates for gradient magnitude changes at high temperature.

Types of Distillation

LLM Distillation

• Alpaca/Vicuna approach: Generate training data from GPT-4 → fine-tune smaller model.
• Not classic distillation (no soft labels) — actually data distillation or imitation learning.
• Logit distillation: Access to teacher logits for each token → train student to match distribution.
• DistilBERT: 40% smaller, 60% faster, retains 97% of BERT performance.
• TinyLlama: 1.1B model trained on same data as larger models — competitive performance.

Practical Guidelines

• Teacher-student size gap: Student should be 2-10x smaller. Too large a gap reduces distillation effectiveness.
• Temperature: Start with T=4, tune in range [2, 20].
• Feature distillation: Add projection layers if teacher/student feature dimensions differ.
• Ensemble teachers: Distilling from an ensemble of teachers gives better results than a single teacher.

Knowledge distillation is the primary technique for deploying large models in resource-constrained environments — from compressing BERT for mobile deployment to creating smaller LLMs from GPT-class teachers, distillation bridges the gap between research-scale accuracy and production-scale efficiency.