Knowledge distillation trains a smaller student model to mimic a larger teacher model, transferring learned knowledge. Core idea: Teacher produces soft probability distributions over outputs. Student learns to match these distributions, not just hard labels. Why soft labels: Contain more information than class. P(cat)=0.7, P(dog)=0.2 tells student about similarity. Dark knowledge. Loss function: KL divergence between student and teacher output distributions (at temperature T), often combined with standard cross-entropy on labels. Temperature: Higher T (e.g., 4-20) softens distributions, exposes more teacher knowledge. Lower for inference. Applications: Create smaller deployment models, ensemble compression, model acceleration, cross-architecture transfer. For LLMs: Distill large LLM into smaller one. Used for Alpaca, Vicuna (learned from GPT outputs). Self-distillation: Model teaches itself from previous checkpoints. Can improve without external teacher. Feature distillation: Match intermediate representations, not just outputs. Supervised vs unsupervised: Can distill on labeled data or unlabeled data (teacher provides labels). Best practices: Temperature tuning important, combine with hard labels, consider intermediate layers.