knowledge distillation

**Knowledge Distillation** — a model compression technique where a small "student" network learns to mimic the behavior of a large "teacher" network, achieving near-teacher accuracy at a fraction of the size. **How It Works** 1. Train a large, accurate teacher model 2. Run teacher on training data → collect "soft labels" (probability distributions, not just the predicted class) 3. Train student to match both: - Hard labels (ground truth) - Soft labels from teacher (with temperature scaling) **Why Soft Labels?** - Hard label: [0, 0, 1, 0] — "this is a cat" - Soft label: [0.01, 0.05, 0.90, 0.04] — "this is mostly cat, slightly dog-like" - Soft labels encode "dark knowledge" — relationships between classes that hard labels miss **Temperature Scaling** $$p_i = \frac{\exp(z_i / T)}{\sum \exp(z_j / T)}$$ - $T > 1$: Softens the distribution (reveals more structure) - Typical: $T = 3$–$20$ during distillation **Results** - Student (1/10th the size) often achieves 95-99% of teacher accuracy - DistilBERT: 60% smaller, 60% faster, retains 97% of BERT's performance - Used in deploying LLMs to mobile/edge devices **Distillation** is one of the most practical compression techniques — it's how large AI models get deployed to real-world applications.