Knowledge Distillation is the model compression technique where a smaller "student" network is trained to mimic the behavior of a larger, more capable "teacher" network — transferring the teacher's learned knowledge through soft probability distributions (soft labels) rather than hard ground-truth labels, enabling the student to achieve accuracy approaching the teacher's while being 3-10x smaller and faster at inference.

Why Soft Labels Carry More Information

A hard label for a cat image is simply [1, 0, 0, ...]. The teacher's soft output might be [0.85, 0.10, 0.03, 0.02, ...] — revealing that this cat slightly resembles a dog, less so a fox, even less a rabbit. These inter-class relationships (dark knowledge) provide richer training signal than hard labels alone. The student learns the teacher's similarity structure over the entire output space, not just the correct class.

Distillation Loss

The standard distillation objective combines soft-label and hard-label losses:

L = α × KL(σ(z_t/T), σ(z_s/T)) × T² + (1-α) × CE(y, σ(z_s))

Where z_t and z_s are teacher and student logits, T is the temperature (typically 3-20) that softens probability distributions, σ is softmax, KL is Kullback-Leibler divergence, CE is cross-entropy with ground truth y, and α balances the two terms. Higher temperature reveals more of the teacher's inter-class knowledge.

Distillation Approaches

• Response-Based (Logit Distillation): Student mimics teacher's output distribution. The original Hinton et al. (2015) formulation. Simple and effective.
• Feature-Based (Hint Learning): Student mimics the teacher's intermediate feature maps, not just outputs. FitNets train the student's hidden layers to match the teacher's using auxiliary regression losses. Transfers structural knowledge about internal representations.
• Relation-Based: Student preserves the relational structure between samples as learned by the teacher — the distance/similarity matrix between all pairs of examples in a batch. Captures holistic structural knowledge.
• Self-Distillation: A model distills into itself — using its own soft predictions (from a previous training epoch, a deeper exit, or an ensemble of augmented views) as targets. Born-Again Networks show that self-distillation improves accuracy without a separate teacher.

LLM Distillation

Distillation is critical for deploying large language models:

• DistilBERT: 6-layer student trained from 12-layer BERT teacher. Retains 97% of BERT's accuracy at 60% the size and 2x speed.
• LLM-to-SLM: Frontier models (GPT-4, Claude) used as teachers to generate training data for smaller models. The teacher's chain-of-thought reasoning is distilled into the student's training corpus.
• Speculative Decoding: A small draft model generates candidate tokens that the large model verifies — combining the speed of the small model with the quality of the large model.

Knowledge Distillation is the bridge between model capability and deployment practicality — extracting the essential learned knowledge from computationally expensive models into efficient ones that can run on mobile devices, edge hardware, and latency-constrained production environments.