model distillation knowledge

**Knowledge Distillation** is the **model compression technique where a smaller "student" network is trained to mimic the behavior of a larger, more capable "teacher" network — transferring the teacher's learned knowledge through soft probability distributions (soft labels) rather than hard ground-truth labels, enabling the student to achieve accuracy approaching the teacher's while being 3-10x smaller and faster at inference**. **Why Soft Labels Carry More Information** A hard label for a cat image is simply [1, 0, 0, ...]. The teacher's soft output might be [0.85, 0.10, 0.03, 0.02, ...] — revealing that this cat slightly resembles a dog, less so a fox, even less a rabbit. These inter-class relationships (dark knowledge) provide richer training signal than hard labels alone. The student learns the teacher's similarity structure over the entire output space, not just the correct class. **Distillation Loss** The standard distillation objective combines soft-label and hard-label losses: L = α × KL(σ(z_t/T), σ(z_s/T)) × T² + (1-α) × CE(y, σ(z_s)) Where z_t and z_s are teacher and student logits, T is the temperature (typically 3-20) that softens probability distributions, σ is softmax, KL is Kullback-Leibler divergence, CE is cross-entropy with ground truth y, and α balances the two terms. Higher temperature reveals more of the teacher's inter-class knowledge. **Distillation Approaches** - **Response-Based (Logit Distillation)**: Student mimics teacher's output distribution. The original Hinton et al. (2015) formulation. Simple and effective. - **Feature-Based (Hint Learning)**: Student mimics the teacher's intermediate feature maps, not just outputs. FitNets train the student's hidden layers to match the teacher's using auxiliary regression losses. Transfers structural knowledge about internal representations. - **Relation-Based**: Student preserves the relational structure between samples as learned by the teacher — the distance/similarity matrix between all pairs of examples in a batch. Captures holistic structural knowledge. - **Self-Distillation**: A model distills into itself — using its own soft predictions (from a previous training epoch, a deeper exit, or an ensemble of augmented views) as targets. Born-Again Networks show that self-distillation improves accuracy without a separate teacher. **LLM Distillation** Distillation is critical for deploying large language models: - **DistilBERT**: 6-layer student trained from 12-layer BERT teacher. Retains 97% of BERT's accuracy at 60% the size and 2x speed. - **LLM-to-SLM**: Frontier models (GPT-4, Claude) used as teachers to generate training data for smaller models. The teacher's chain-of-thought reasoning is distilled into the student's training corpus. - **Speculative Decoding**: A small draft model generates candidate tokens that the large model verifies — combining the speed of the small model with the quality of the large model. Knowledge Distillation is **the bridge between model capability and deployment practicality** — extracting the essential learned knowledge from computationally expensive models into efficient ones that can run on mobile devices, edge hardware, and latency-constrained production environments.