knowledge distillation

**Knowledge Distillation** is the **model compression technique where a large, high-accuracy "teacher" model transfers its learned knowledge to a smaller, faster "student" model by training the student to match the teacher's soft probability outputs rather than the hard ground-truth labels — capturing the dark knowledge encoded in the teacher's inter-class similarity structure**. **Why Soft Labels Carry More Information Than Hard Labels** A hard label says "this is a cat" (one-hot: [0, 0, 1, 0]). The teacher's soft output says "this is 85% cat, 10% lynx, 4% dog, 1% horse." The 10% lynx probability encodes the teacher's knowledge that cats and lynxes share visual features — information completely absent from the hard label. By learning from soft targets, the student acquires structural knowledge about the relationships between classes that would require far more data to learn from hard labels alone. **The Distillation Framework** - **Temperature Scaling**: The teacher's logits are divided by a temperature parameter T before softmax. Higher T produces softer (more uniform) distributions, amplifying the dark knowledge in the tail probabilities. Typical values range from T=2 to T=20. - **Loss Function**: The student minimizes a weighted combination of cross-entropy with ground truth labels and KL divergence with the teacher's soft predictions. A T-squared correction factor adjusts for the gradient magnitude change under temperature scaling. - **Feature Distillation**: Beyond output logits, the student can be trained to match the teacher's intermediate feature representations (FitNets, attention maps, CKA-aligned hidden states). This provides richer supervision for student architectures that differ substantially from the teacher. **Distillation in Practice** - **LLM Distillation**: A 70B teacher generates training data (prompt-completion pairs) and soft logits. A 7B student trained on this data often outperforms a 7B model trained directly on the same raw corpus, because the teacher's outputs provide a stronger, denoised training signal. - **On-Policy Distillation**: The student generates its own completions, and the teacher scores them. This trains the student on its own output distribution, avoiding the distribution mismatch of training on the teacher's completions. - **Self-Distillation**: A model distills knowledge into itself — an earlier checkpoint or a pruned version. Even without a capacity difference, self-distillation consistently improves calibration and generalization. **Limitations** Distillation quality is bounded by the teacher's accuracy on the target domain. A teacher that struggles on medical text will not produce useful soft labels for a medical student model. Teacher errors are inherited by the student, sometimes amplified. Knowledge Distillation is **the most reliable technique for shipping large-model intelligence in small-model form factors** — compressing months of teacher training compute into a student that runs on a mobile device or edge accelerator.