Knowledge Distillation Teacher-Student Learning
# Knowledge Distillation: Teacher-Student Learning
## Introduction & Motivation
Knowledge distillation transfers knowledge from large teacher model to small student model. Soft targets: teacher's probability distributions guide student. Maintains performance while reducing parameters/latency. Applications: model compression, mobile deployment, ensemble distillation.
Motivation: Large models impractical for deployment. Distillation compresses knowledge into smaller footprint without retraining from scratch.
Applications: Mobile inference, edge deployment, real-time prediction, ensemble compression.
---
## Core Concepts & Theory
### Soft Targets
Teacher output probabilities (soft labels) encode class relationships. Temperature T controls softness; higher T = softer distribution.
### Distillation Loss
Weighted combination: α × CE(student, hard labels) + (1-α) × KL(soft teacher, soft student).
### Transfer Effectiveness
What transfers: class similarity structure, feature representations, reasoning patterns.
---
## Mathematical Formulation
Soft target probability (temperature-scaled):
$$q_i = \frac{\exp(z_i^T / T)}{\sum_j \exp(z_j^T / T)}$$
Distillation loss:
$$\mathcal{L}_{ ext{dist}} = \alpha H(y, \sigma(z_S)) + (1-\alpha) \cdot T^2 \cdot D_{ ext{KL}}(q^T || q^S)$$
where q^T, q^S = teacher/student soft probs.
---
## Advanced Theory & Extensions
### Feature-Based Distillation
Match intermediate layer representations; attention transfer.
### Multi-Teacher Distillation
Ensemble teachers guide student; diverse knowledge.
### Online Distillation
Co-training; students teach each other.
---
## Computational Considerations
Forward pass: O(1) per sample (teacher + student).
Teacher inference cost: Typically amortized offline.
Memory: Both models in memory; trade training time for smaller student.
---
## Practical Implementation Strategies
### Temperature Selection
T ∈ [1, 20]; higher = softer, helps learning. Typically 5-10.
### Loss Weight Balancing
α ∈ [0.1, 0.9]; balance hard/soft supervision. Often α=0.5.
### Layer Matching
Select intermediate layers for feature distillation; symmetric sizes.
---
## Benchmark Datasets & Evaluation
ImageNet: Teacher ResNet-152 → Student ResNet-50; minimal accuracy drop.
CIFAR-10: Small networks; compression ratio 4-10×.
Metrics: Student accuracy, model size, inference latency.
---
## Key Challenges & Limitations
### Teacher Quality
Poor teacher → poor student. Teacher must be significantly better.
### Temperature Tuning
Sensitive hyperparameter; requires grid search.
### Architecture Mismatch
Vastly different student architecture reduces transfer effectiveness.
---
## Hyperparameter Tuning
Temperature T: 3-20; task-dependent.
Distillation weight α: 0.3-0.7.
Teacher/student capacity ratio: 2-10×.
---
## Real-World Applications & Case Studies
BERT Distillation: DistilBERT achieves 97% BERT performance, 40% smaller.
MobileNets: Distilled from larger architectures; optimized for mobile.
Ensemble Compression: Ensemble of 10 models → single student model.
---
## Integration with Other Methods
Distillation + Quantization → compress further via bit reduction.
Distillation + Pruning → combine knowledge transfer with sparsity.
---
## Summary & Key Takeaways
Knowledge distillation transfers soft knowledge (probability distributions) from teacher to student, enabling model compression while maintaining performance.
Principles:
1. Soft targets encode class relationships.
2. Temperature controls softness of distribution.
3. Loss combines supervised + KL divergence terms.
4. Teacher quality critical; must significantly outperform student.
5. Feature-level distillation adds alignment constraints.
---
---
## Appendix: Practical Labs
### Lab 1: Soft Target Generation
import torch
import torch.nn.functional as F
import numpy as np
def compute_soft_targets(teacher_logits, temperature=5.0):
"""Compute soft probability targets from teacher"""
return F.softmax(teacher_logits / temperature, dim=1)
# Test
logits_teacher = torch.randn(32, 10)
soft_targets = compute_soft_targets(logits_teacher, temperature=5.0)
print(f"Soft targets shape: {soft_targets.shape}")
assert soft_targets.shape == (32, 10), "Should preserve batch/class dims"
assert torch.allclose(soft_targets.sum(dim=1), torch.ones(32), atol=1e-5), "Should sum to 1"
assert (soft_targets >= 0).all(), "Probabilities should be non-negative"
print("✓ Soft target generation working")
if __name__ == "__main__":
print("Lab 1: Soft Targets - PASSED")### Lab 2: Distillation Loss
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
def distillation_loss(student_logits, teacher_logits, labels, temperature=5.0, alpha=0.5):
"""Combined supervised + distillation loss"""
# Hard loss: cross-entropy on student
hard_loss = F.cross_entropy(student_logits, labels)
# Soft loss: KL divergence
soft_student = F.log_softmax(student_logits / temperature, dim=1)
soft_teacher = F.softmax(teacher_logits / temperature, dim=1)
soft_loss = F.kl_div(soft_student, soft_teacher, reduction='mean')
# Combined
loss = alpha * hard_loss + (1 - alpha) * (temperature ** 2) * soft_loss
return loss
# Test
student_logits = torch.randn(32, 10)
teacher_logits = torch.randn(32, 10)
labels = torch.randint(0, 10, (32,))
loss = distillation_loss(student_logits, teacher_logits, labels, temperature=5.0, alpha=0.5)
print(f"Distillation loss: {loss:.4f}")
assert 0 < loss < 100, "Loss should be reasonable"
assert np.isfinite(loss.item()), "Loss should be finite"
print("✓ Distillation loss working")
if __name__ == "__main__":
print("Lab 2: Distillation Loss - PASSED")### Lab 3: Teacher-Student Training Step
import torch
import torch.nn as nn
import torch.optim as optim
class SimpleNet(nn.Module):
def __init__(self, in_dim=10, hidden=32, out_dim=5):
super().__init__()
self.net = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU(), nn.Linear(hidden, out_dim))
def forward(self, x):
return self.net(x)
# Models
teacher = SimpleNet(in_dim=10, hidden=64, out_dim=5)
student = SimpleNet(in_dim=10, hidden=16, out_dim=5)
teacher.eval() # Teacher frozen
student_opt = optim.Adam(student.parameters(), lr=0.01)
# Data
x = torch.randn(32, 10)
y = torch.randint(0, 5, (32,))
# Training step
with torch.no_grad():
teacher_logits = teacher(x)
student_logits = student(x)
loss = nn.functional.cross_entropy(student_logits, y)
student_opt.zero_grad()
loss.backward()
student_opt.step()
print(f"Training loss: {loss:.4f}")
assert loss > 0, "Loss should be positive"
print("✓ Teacher-student training working")
if __name__ == "__main__":
print("Lab 3: Training - PASSED")### Lab 4: Model Compression Evaluation
import torch
import torch.nn as nn
def count_parameters(model):
"""Count total parameters"""
return sum(p.numel() for p in model.parameters())
def evaluate_compression(teacher, student, x_test, y_test):
"""Evaluate compression: accuracy trade-off vs size reduction"""
teacher.eval()
student.eval()
with torch.no_grad():
teacher_pred = (teacher(x_test).argmax(dim=1) == y_test).float().mean()
student_pred = (student(x_test).argmax(dim=1) == y_test).float().mean()
teacher_params = count_parameters(teacher)
student_params = count_parameters(student)
compression_ratio = teacher_params / student_params
accuracy_ratio = student_pred / teacher_pred
return compression_ratio, accuracy_ratio
# Test
teacher = nn.Sequential(nn.Linear(20, 128), nn.ReLU(), nn.Linear(128, 10))
student = nn.Sequential(nn.Linear(20, 32), nn.ReLU(), nn.Linear(32, 10))
x_test = torch.randn(50, 20)
y_test = torch.randint(0, 10, (50,))
compression, acc_ratio = evaluate_compression(teacher, student, x_test, y_test)
print(f"Compression ratio: {compression:.2f}×, Accuracy ratio: {acc_ratio:.2%}")
assert compression > 1, "Student should be smaller"
assert 0 < acc_ratio <= 1, "Accuracy ratio should be valid"
print("✓ Compression evaluation working")
if __name__ == "__main__":
print("Lab 4: Compression - PASSED")