Knowledge distillation is a model-compression technique in which a small, cheap "student" model is trained to reproduce the behavior of a large, accurate "teacher" model. Instead of training the student only on the correct answers, you train it to match the teacher's full output — its entire probability distribution over possible answers. The result is a compact model that runs far faster and cheaper than the teacher while retaining much of its quality. Distillation is one of the main ways a frontier-scale model gets turned into something small enough to deploy at scale or on-device.\n\n``svg\n\n``\n\nThe key insight is that soft labels carry more information than hard labels. A one-hot training label says only "the answer is cat." The teacher's output says "92% cat, 5% dog, 2% fox, 0.1% car" — and those small non-zero probabilities, sometimes called dark knowledge, tell the student which wrong answers are reasonable and which are absurd. Learning from this richer signal lets a small model absorb structure it could never discover from hard labels alone, which is why a distilled student often beats a same-size model trained from scratch.\n\nTemperature softens the distribution so the student can see it. A confident teacher puts nearly all its probability on one class, hiding the informative tail. Raising the softmax temperature spreads the distribution out, exaggerating the relative sizes of the small probabilities so the student can learn from them. The student is trained with the same temperature, typically against a blend of two losses: matching the teacher's soft labels and still getting the true hard label right.\n\nDistillation buys efficiency, not new capability. The student cannot exceed the teacher on the teacher's own task — it is imitating a ceiling. What it gains is dramatically lower inference cost: fewer parameters, less memory, lower latency, and lower energy per query. For high-volume serving or edge deployment, a student that keeps most of the teacher's accuracy at a fraction of the cost is an enormous practical win.\n\nIt comes in several flavors. Response-based distillation matches final output probabilities (the classic form). Feature-based distillation also matches intermediate hidden representations, giving the student a richer target. Self-distillation trains a model from an earlier copy of itself, and online distillation trains teacher and student together. In modern LLMs, a common pattern is to have a large model generate high-quality outputs and then fine-tune a smaller model on them — effectively distillation through generated data.\n\nIt pairs naturally with quantization and pruning. Distillation reduces the number of parameters or the architecture size; quantization reduces the precision of each parameter; pruning removes unimportant weights. They are complementary and routinely stacked — distill to a smaller architecture, then quantize it to low precision — to hit aggressive latency and memory budgets for deployment.\n\n| Aspect | Teacher | Student (distilled) |\n|---|---|---|\n| Size | large | small |\n| Accuracy | highest | close to teacher, below it |\n| Inference cost | high | low |\n| Trained on | data + hard labels | teacher's soft labels (+ hard labels) |\n| Role | quality reference | deployable workhorse |\n\nRead distillation through an imitation-transfer lens rather than a shrink-the-file lens: you are not compressing weights, you are transferring behavior. The teacher's soft, full-distribution outputs are a far more informative teaching signal than raw labels, and that signal is what lets a small model punch above its size — capturing most of a giant model's competence at a small fraction of its running cost, which is exactly what makes large models economical to actually deploy.\n