what is model distillation
Model distillation is a technique for training a smaller, faster "student" AI model to mimic the behavior of a larger, more capable "teacher" model, transferring much of the teacher's learned knowledge into a compact form that can run far more efficiently, often with only a modest reduction in accuracy.
```flowchart
{
"rows": [
{ "type": "nodes", "items": [
{ "title": "Large, capable \"teacher\" model already trained", "sub": "high accuracy, but large and slow to run", "tone": "blue" }
]},
{ "type": "arrow" },
{ "type": "group", "title": "Smaller \"student\" model trained to mimic it", "items": [
{ "title": "Learns from the teacher's outputs and behavior", "sub": "not just the original training data alone", "tone": "green" }
]},
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "Compact model, most of the capability", "sub": "much cheaper and faster to run", "tone": "orange" }
]}
]
}
```
**Distillation works because a large model's own outputs carry richer information than the raw training labels alone.** Rather than training the smaller student model only on the same basic correct-answer labels the teacher was originally trained on, distillation typically has the student learn from the teacher's full output patterns — including how confident it is across various possible answers — a richer training signal that helps the smaller model learn more efficiently than training from scratch on the original data alone.
```svg
```
| Aspect | Teacher model | Distilled student model |
|---|---|---|
| Size | Large | Much smaller |
| Accuracy | Highest achievable | Slightly lower, often close |
| Compute cost to run | High | Significantly lower |
| Typical role | Training-time reference | Deployed in production, especially resource-constrained settings |
**Distillation and quantization solve related but distinct problems, and are frequently used together for maximum efficiency.** Quantization reduces the numerical precision of an existing model's weights, while distillation trains an entirely new, smaller model architecture to mimic a larger one's behavior — these are complementary techniques, and a heavily optimized deployed model often uses both: first distilled down to a smaller architecture, then further quantized to reduce its numerical precision.
**A well-executed distillation can preserve a surprising amount of the teacher model's practical capability despite a dramatic reduction in size.** Because the student model learns from the teacher's nuanced output patterns rather than starting from nothing, a properly distilled student can often retain most of the teacher's performance on the tasks it was distilled for, even at a small fraction of the parameter count — though the student typically doesn't match the teacher's performance on tasks or edge cases well outside what it was specifically trained to handle.
**Distillation is a major reason many AI products can serve fast, affordable responses to enormous numbers of users despite being built on top of far larger underlying models.** Running a full-size, maximally capable model for every single user request would often be prohibitively slow and expensive at large scale, so many production AI systems rely on a distilled, more efficient version of a larger model for everyday use — reserving the full, larger model for training or for cases where its extra capability is genuinely needed.
Read model distillation through a teacher-and-student lens: rather than training a small model from scratch and hoping it learns as much as a much larger model did, distillation has the smaller model learn directly from a more capable one's behavior, transferring much of that hard-won capability into a form efficient enough to actually deploy at scale.