Knowledge Distillation is the training paradigm where smaller student networks learn from larger teacher model via soft target distributions — achieving substantial parameter reduction (40%+ compression) with minimal performance loss, enabling deployment on edge/mobile devices.

Core Distillation Framework:

• Teacher-student training: large pretrained teacher model guides smaller student network; knowledge transfer via probability distributions
• Soft targets: teacher output softmax logits (before argmax) preserve uncertainty and inter-class relationships compared to hard labels
• Distillation loss: combines student cross-entropy on original labels with KL divergence from teacher soft targets
• Temperature scaling: softening both teacher and student outputs via Softmax(z/T); higher T (4-10) creates gentler probability landscape for learning

Hinton's Distillation Objective:

• KL divergence loss: measures divergence between student and teacher probability distributions; guides student learning beyond hard labels
• Weighted combination: total loss = α·cross_entropy(student, hard_labels) + (1-α)·KL_divergence(student, teacher_soft_targets)
• Temperature effect: Softmax(z/T) for T > 1 reduces peaks in distributions; smoother gradients aid student learning
• Optimal temperature: typically 3-20 depending on dataset; higher T for more complex task knowledge transfer

Knowledge Distillation Variants:

• Response-based distillation: student matches final output distribution; most common; effective for classification tasks
• Feature-based distillation: intermediate layer feature matching between student/teacher; additional guidance beyond final output
• Relation-based distillation: student learns relationships between data examples from teacher; meta-knowledge transfer

DistilBERT and TinyBERT:

• DistilBERT: 40% parameter reduction from BERT (66M vs 110M), 60% speedup, 97% GLUE performance retention
• Distillation + pruning + quantization: combined compression techniques achieve 2-4x speedup with minimal quality loss
• TinyBERT: task-specific distillation for further compression; knowledge distillation at intermediate layers
• Applications: mobile inference, edge deployment, real-time inference with resource constraints

Deployment Benefits:

• Smaller model size: enables deployment on smartphones, IoT devices, browser-based ML
• Inference latency reduction: fewer parameters, smaller memory footprint → faster inference
• Energy efficiency: reduced computation and memory bandwidth crucial for battery-powered devices
• Cost reduction: fewer parameters → cheaper inference infrastructure

Knowledge distillation successfully transfers learned representations from large teacher models to compact students — enabling efficient deployment without significant performance degradation across NLP and vision tasks.