knowledge distillation teacher student

**Knowledge Distillation** is the **training paradigm where smaller student networks learn from larger teacher model via soft target distributions — achieving substantial parameter reduction (40%+ compression) with minimal performance loss, enabling deployment on edge/mobile devices**. **Core Distillation Framework:** - Teacher-student training: large pretrained teacher model guides smaller student network; knowledge transfer via probability distributions - Soft targets: teacher output softmax logits (before argmax) preserve uncertainty and inter-class relationships compared to hard labels - Distillation loss: combines student cross-entropy on original labels with KL divergence from teacher soft targets - Temperature scaling: softening both teacher and student outputs via Softmax(z/T); higher T (4-10) creates gentler probability landscape for learning **Hinton's Distillation Objective:** - KL divergence loss: measures divergence between student and teacher probability distributions; guides student learning beyond hard labels - Weighted combination: total loss = α·cross_entropy(student, hard_labels) + (1-α)·KL_divergence(student, teacher_soft_targets) - Temperature effect: Softmax(z/T) for T > 1 reduces peaks in distributions; smoother gradients aid student learning - Optimal temperature: typically 3-20 depending on dataset; higher T for more complex task knowledge transfer **Knowledge Distillation Variants:** - Response-based distillation: student matches final output distribution; most common; effective for classification tasks - Feature-based distillation: intermediate layer feature matching between student/teacher; additional guidance beyond final output - Relation-based distillation: student learns relationships between data examples from teacher; meta-knowledge transfer **DistilBERT and TinyBERT:** - DistilBERT: 40% parameter reduction from BERT (66M vs 110M), 60% speedup, 97% GLUE performance retention - Distillation + pruning + quantization: combined compression techniques achieve 2-4x speedup with minimal quality loss - TinyBERT: task-specific distillation for further compression; knowledge distillation at intermediate layers - Applications: mobile inference, edge deployment, real-time inference with resource constraints **Deployment Benefits:** - Smaller model size: enables deployment on smartphones, IoT devices, browser-based ML - Inference latency reduction: fewer parameters, smaller memory footprint → faster inference - Energy efficiency: reduced computation and memory bandwidth crucial for battery-powered devices - Cost reduction: fewer parameters → cheaper inference infrastructure **Knowledge distillation successfully transfers learned representations from large teacher models to compact students — enabling efficient deployment without significant performance degradation across NLP and vision tasks.**