knowledge distillation advanced
**Advanced Knowledge Distillation** encompasses **sophisticated techniques for transferring knowledge from a large teacher model to a smaller student model beyond basic soft-label matching** — including intermediate feature distillation, attention transfer, relational knowledge distillation, and task-specific distillation strategies that enable students to capture structural knowledge the teacher has learned, not just its output predictions.
**Beyond Basic Logit Distillation**
Basic KD (Hinton 2015) matches the student's softmax output to the teacher's soft labels. Advanced methods distill knowledge from multiple levels:
```svg
```
**Feature/Intermediate Layer Distillation**
| Method | What is Distilled | Loss |
|--------|------------------|------|
| FitNets | Hidden layer activations | MSE(student_feat, teacher_feat) via adapter |
| Attention Transfer (AT) | Attention maps (spatial) | MSE on attention map norm |
| PKT (Probabilistic KT) | Feature distribution in embedding space | KL divergence |
| NST (Neuron Selectivity) | Neuron activation distributions | MMD (Maximum Mean Discrepancy) |
| CRD (Contrastive Rep. Dist.) | Representation structure | Contrastive loss |
Adapter layers (1×1 conv or linear projection) bridge dimension mismatches between teacher and student hidden layers.
**Relational Knowledge Distillation (RKD)**
Instead of matching individual outputs, RKD transfers the **relationships** between samples:
```python
# Distance-wise RKD: preserve pairwise distance structure
teacher_dist = pairwise_distance(teacher_embeddings) # NxN matrix
student_dist = pairwise_distance(student_embeddings)
loss_rkd = huber_loss(student_dist / student_dist.mean(),
teacher_dist / teacher_dist.mean())
# Angle-wise RKD: preserve angular relationships among triplets
# Captures higher-order structural information
```
**LLM-Specific Distillation**
Distilling large language models has unique considerations:
- **Black-box distillation**: When teacher weights are inaccessible (GPT-4 → open model), use only generated outputs. Techniques: instruction following data generation, chain-of-thought distillation (distill reasoning traces, not just answers).
- **White-box distillation**: With teacher weight access, match logit distributions over the full vocabulary (50K+ dimension KL divergence) and intermediate transformer layer representations.
- **Progressive distillation**: Gradually reduce model size through multiple distillation stages rather than one large compression step.
- **Distillation for specific capabilities**: Selectively distill math reasoning, code generation, or instruction following by curating task-specific transfer sets.
**Multi-Teacher and Self-Distillation**
- **Multi-teacher**: Ensemble of specialists, each contributing expertise. Student learns from domain-weighted combination of teacher outputs.
- **Self-distillation**: Model distills knowledge from its own deeper layers to shallower layers, or from a previous training epoch to the current one.
- **Born-Again Networks**: Iteratively distill — student becomes the new teacher for the next round, often surpassing the original teacher.
**Advanced knowledge distillation is the primary model compression technique enabling deployment of LLM-class intelligence on resource-constrained devices** — by transferring not just predictions but structural, relational, and intermediate representations, modern distillation achieves compression ratios of 10-100× while retaining 90-98% of teacher performance.