knowledge distillation training

**Knowledge Distillation** is **the model compression technique where a large, high-performing teacher model transfers its learned representations to a smaller, more efficient student model — training the student to mimic the teacher's soft probability distributions rather than just the hard ground-truth labels, enabling the student to capture inter-class relationships and decision boundaries that hard labels cannot convey**. **Distillation Framework:** - **Soft Labels**: teacher's output probabilities (after softmax) contain rich information; for a cat image, the teacher might output [cat: 0.85, dog: 0.10, fox: 0.04, ...] — these relative probabilities tell the student that cats look somewhat like dogs, which hard one-hot labels [cat: 1, rest: 0] cannot express - **Temperature Scaling**: softmax temperature T controls the entropy of the teacher's output distribution; higher T (2-20) softens the distribution, making small probabilities more visible; distillation loss uses temperature T; inference uses T=1 - **Combined Loss**: student minimizes α·KL(teacher_soft, student_soft) + (1-α)·CE(ground_truth, student_hard); typical α=0.5-0.9; the soft label loss provides the teacher's dark knowledge while the hard label loss anchors to ground truth - **Offline vs Online**: offline distillation pre-computes teacher outputs for the entire dataset; online distillation runs teacher and student simultaneously, allowing the teacher to continue improving during distillation **Distillation Strategies:** - **Logit Distillation (Hinton)**: student matches teacher's final softmax output distribution; simplest and most common; effective for classification tasks but loses intermediate feature information - **Feature Distillation (FitNets)**: student matches teacher's intermediate feature maps at selected layers; requires adaptation layers (1×1 convolutions) when teacher and student have different channel dimensions; captures richer representational knowledge than logit-only distillation - **Attention Transfer**: student matches teacher's attention maps (spatial or channel attention patterns); forces the student to focus on the same regions as the teacher — particularly effective for vision models - **Relational Distillation**: student preserves the relationships between sample representations (e.g., pairwise distances or angles in embedding space) rather than matching individual outputs — captures structural knowledge invariant to representation scale **Advanced Techniques:** - **Self-Distillation**: model distills knowledge from its own deeper layers to shallower layers, or from later training epochs to earlier epochs; no separate teacher required; improves accuracy by 1-3% on image classification - **Multi-Teacher Distillation**: ensemble of diverse teacher models provides averaged or combined soft labels; student learns from the collective knowledge of multiple specialists; ensemble agreement regions receive stronger teaching signal - **Progressive Distillation**: chain of progressively smaller students, each distilling from the previous one rather than directly from the large teacher; bridges large capacity gaps that single-step distillation struggles with - **Task-Specific Distillation**: for LLMs, distillation on task-specific data (instruction-following, code generation, reasoning) is more efficient than general distillation; DistilBERT, TinyLlama, and Phi models demonstrate task-focused distillation **Results and Applications:** - **Compression Ratios**: typical 4-10× parameter reduction with <2% accuracy loss; DistilBERT achieves 97% of BERT performance with 40% fewer parameters and 60% faster inference - **Cross-Architecture**: teacher and student can have different architectures (CNN teacher → efficient architecture student); knowledge transfers across architecture families - **Deployment**: distilled models deployed on edge devices (phones, embedded systems) where teacher models are too large; enables state-of-the-art accuracy within strict latency and memory budgets Knowledge distillation is **the most practical technique for deploying large model capabilities on resource-constrained hardware — transferring the dark knowledge embedded in teacher probability distributions to compact student models, enabling the accuracy benefits of massive models to reach every device and application**.