Knowledge Distillation
# Knowledge Distillation
## Introduction & Motivation
Knowledge Distillation: transfer knowledge from large teacher models to smaller student models. Model compression; efficient inference. Applications: mobile deployment, real-time systems, edge devices.
Motivation: Reduce model size and inference time while maintaining performance.
Applications: Mobile models, edge deployment, real-time inference.
---
## Core Concepts & Theory
### Teacher-Student Framework
Large teacher guides smaller student.
### Knowledge Transfer
Soft targets via teacher probabilities.
### Dark Knowledge
Capturing teacher's decision boundaries.
### Attention Transfer
Transfer attention maps between models.
---
## Mathematical Formulation
Distillation Loss:
$$L = \alpha L_ ext{CE}(y, \hat{y}) + (1-\alpha) L_ ext{KL}(p_S, p_T)$$
Temperature Scaling:
$$p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$$
KL Divergence Loss:
$$L_ ext{KL} = \sum_i p_T(i) \log \frac{p_T(i)}{p_S(i)}$$
---
## Advanced Theory & Extensions
### Attention Transfer
Transfer attention maps from teacher.
### Relational Knowledge Distillation
Transfer inter-sample relationships.
### Multi-Teacher Distillation
Learn from multiple teachers.
---
## Computational Considerations
Teacher inference: O(model_size_teacher).
Student inference: O(model_size_student).
Distillation training: O(student·iterations).
---
## Practical Implementation Strategies
### Temperature Selection
Typically 3-20 for soft targets.
### Loss Weighting
Balance CE and KL losses (50-50 or 90-10).
### Teacher Architecture
Mismatch between teacher and student.
---
## Benchmark Datasets & Evaluation
ImageNet: Classification task.
CIFAR-10/100: Smaller scale benchmarks.
Standard datasets with size-performance curves.
---
## Key Challenges & Limitations
### Architecture Mismatch
Different capacities between models.
### Transfer Specificity
Task and domain dependence.
### Dark Knowledge Extraction
Complex knowledge in soft targets.
---
## Hyperparameter Tuning
Temperature: 3-20.
Knowledge weight (α): 0.1-0.9.
Learning rate: 1x-10x student-only rate.
---
## Real-World Applications & Case Studies
Mobile CNNs: Smaller models for smartphones.
BERT Distillation: DistilBERT for efficient NLP.
Edge Deployment: Real-time object detection.
---
## Integration with Other Methods
Knowledge distillation + pruning for extreme compression; + quantization for further speedup.
---
## Summary & Key Takeaways
Knowledge Distillation via teacher-student training enables efficient model compression.
Principles:
1. Soft targets: Dark knowledge transfer.
2. Temperature scaling: Smooth probabilities.
3. Loss combination: CE + KL.
4. Architecture flexibility: Size mismatch OK.
5. Efficient inference: Smaller student model.
---
---
## Appendix: Practical Labs
### Lab 1: Temperature Scaling
import numpy as np
def temperature_scaled_softmax(logits, temperature=1.0):
"""Apply temperature to logits"""
scaled_logits = logits / temperature
# Softmax
exp_logits = np.exp(scaled_logits - np.max(scaled_logits, axis=1, keepdims=True))
probs = exp_logits / np.sum(exp_logits, axis=1, keepdims=True)
return probs
# Test
np.random.seed(42)
logits = np.random.randn(4, 10)
probs_low = temperature_scaled_softmax(logits, temperature=0.5)
probs_high = temperature_scaled_softmax(logits, temperature=5.0)
assert probs_low.shape == (4, 10), "Shape preserved"
assert np.allclose(probs_low.sum(axis=1), 1), "Sums to 1"
print("✓ Temperature scaling working")
if __name__ == "__main__":
print("Lab 1: TemperatureScaling - PASSED")### Lab 2: KL Divergence Loss
import numpy as np
def kl_divergence_loss(student_probs, teacher_probs):
"""Compute KL divergence between student and teacher"""
# Clip to avoid log(0)
student_probs = np.clip(student_probs, 1e-7, 1)
teacher_probs = np.clip(teacher_probs, 1e-7, 1)
kl_loss = np.sum(teacher_probs * (np.log(teacher_probs) - np.log(student_probs)))
return kl_loss
# Test
np.random.seed(42)
student = np.random.rand(1, 10)
student = student / student.sum(axis=1, keepdims=True)
teacher = np.random.rand(1, 10)
teacher = teacher / teacher.sum(axis=1, keepdims=True)
loss = kl_divergence_loss(student, teacher)
assert loss >= 0, "KL divergence non-negative"
print("✓ KL divergence loss working")
if __name__ == "__main__":
print("Lab 2: KLDivergenceLoss - PASSED")### Lab 3: Distillation Loss
import numpy as np
def distillation_loss(student_logits, teacher_logits, true_labels, temperature=4, alpha=0.5):
"""Combined distillation loss"""
# Temperature scaled softmax
student_probs = np.exp(student_logits / temperature) / np.sum(np.exp(student_logits / temperature), axis=1, keepdims=True)
teacher_probs = np.exp(teacher_logits / temperature) / np.sum(np.exp(teacher_logits / temperature), axis=1, keepdims=True)
# KL divergence
kl_loss = np.sum(teacher_probs * (np.log(teacher_probs) - np.log(student_probs)))
# Cross-entropy
student_hard_probs = np.exp(student_logits) / np.sum(np.exp(student_logits), axis=1, keepdims=True)
ce_loss = -np.mean(np.log(student_hard_probs[np.arange(len(true_labels)), true_labels] + 1e-7))
# Combined
total_loss = alpha * ce_loss + (1 - alpha) * kl_loss
return total_loss
# Test
np.random.seed(42)
student_logits = np.random.randn(4, 10)
teacher_logits = np.random.randn(4, 10)
labels = np.array([0, 1, 2, 3])
loss = distillation_loss(student_logits, teacher_logits, labels)
assert np.isfinite(loss), "Loss finite"
print("✓ Distillation loss working")
if __name__ == "__main__":
print("Lab 3: DistillationLoss - PASSED")### Lab 4: Attention Transfer
import numpy as np
def attention_transfer_loss(student_attention, teacher_attention):
"""Compute attention transfer loss"""
# Normalize attention maps
student_norm = student_attention / (np.sum(student_attention, axis=-1, keepdims=True) + 1e-8)
teacher_norm = teacher_attention / (np.sum(teacher_attention, axis=-1, keepdims=True) + 1e-8)
# MSE loss
loss = np.mean((student_norm - teacher_norm) ** 2)
return loss
# Test
np.random.seed(42)
student_attn = np.random.rand(16, 64, 64)
teacher_attn = np.random.rand(16, 64, 64)
loss = attention_transfer_loss(student_attn, teacher_attn)
assert loss >= 0, "Loss non-negative"
print("✓ Attention transfer working")
if __name__ == "__main__":
print("Lab 4: AttentionTransfer - PASSED")