knowledge distillation

Knowledge distillation trains a smaller student model to mimic a larger teacher model, transferring learned knowledge. **Core idea**: Teacher produces soft probability distributions over outputs. Student learns to match these distributions, not just hard labels. **Why soft labels**: Contain more information than class. P(cat)=0.7, P(dog)=0.2 tells student about similarity. Dark knowledge. **Loss function**: KL divergence between student and teacher output distributions (at temperature T), often combined with standard cross-entropy on labels. **Temperature**: Higher T (e.g., 4-20) softens distributions, exposes more teacher knowledge. Lower for inference. **Applications**: Create smaller deployment models, ensemble compression, model acceleration, cross-architecture transfer. **For LLMs**: Distill large LLM into smaller one. Used for Alpaca, Vicuna (learned from GPT outputs). **Self-distillation**: Model teaches itself from previous checkpoints. Can improve without external teacher. **Feature distillation**: Match intermediate representations, not just outputs. **Supervised vs unsupervised**: Can distill on labeled data or unlabeled data (teacher provides labels). **Best practices**: Temperature tuning important, combine with hard labels, consider intermediate layers.