knowledge distillation
**Knowledge Distillation** is the **model compression technique where a smaller "student" network is trained to mimic the output behavior of a larger, more accurate "teacher" network** — transferring the teacher's learned knowledge through soft probability distributions rather than hard labels, enabling deployment of compact models that retain 90-99% of the teacher's accuracy at a fraction of the size and computation.
**Core Idea (Hinton et al., 2015)**
- Teacher output (softmax with temperature T): $p_i^T = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}$.
- At high temperature (T=4-20): Softmax outputs reveal **inter-class relationships** (e.g., "3" looks more like "8" than like "7").
- These soft labels carry richer information than one-hot hard labels.
- Student learns to match teacher's soft distribution → learns the teacher's reasoning patterns.
**Distillation Loss**
$L = \alpha \cdot T^2 \cdot KL(p^T_{teacher} || p^T_{student}) + (1-\alpha) \cdot CE(y, p_{student})$
- First term: Match teacher's soft predictions (KL divergence).
- Second term: Match ground truth labels (cross-entropy).
- α: Balance between teacher guidance and ground truth (typically 0.5-0.9).
- T²: Compensates for gradient magnitude changes at high temperature.
**Types of Distillation**
| Type | What's Transferred | Example |
|------|-------------------|--------|
| Response-based | Final layer outputs (logits) | Classic Hinton distillation |
| Feature-based | Intermediate layer activations | FitNets, attention transfer |
| Relation-based | Relationships between samples | Relational KD, CRD |
| Self-distillation | Same architecture, deeper→shallower | Born-Again Networks |
| Online distillation | Multiple models teach each other | Deep Mutual Learning |
**LLM Distillation**
- **Alpaca/Vicuna approach**: Generate training data from GPT-4 → fine-tune smaller model.
- Not classic distillation (no soft labels) — actually **data distillation** or **imitation learning**.
- **Logit distillation**: Access to teacher logits for each token → train student to match distribution.
- **DistilBERT**: 40% smaller, 60% faster, retains 97% of BERT performance.
- **TinyLlama**: 1.1B model trained on same data as larger models — competitive performance.
**Practical Guidelines**
- Teacher-student size gap: Student should be 2-10x smaller. Too large a gap reduces distillation effectiveness.
- Temperature: Start with T=4, tune in range [2, 20].
- Feature distillation: Add projection layers if teacher/student feature dimensions differ.
- Ensemble teachers: Distilling from an ensemble of teachers gives better results than a single teacher.
Knowledge distillation is **the primary technique for deploying large models in resource-constrained environments** — from compressing BERT for mobile deployment to creating smaller LLMs from GPT-class teachers, distillation bridges the gap between research-scale accuracy and production-scale efficiency.