Knowledge Distillation Model Compression Lightweight Networks
# Knowledge Distillation: Model Compression & Lightweight Networks
## Introduction & Motivation
Knowledge distillation: transfer knowledge from large to small model. Teacher-student: large model guides small. Distillation loss: match probabilities, features. Applications: mobile deployment, edge computing, latency reduction.
Motivation: Large models accurate but slow. Distillation enables efficient deployment.
Applications: Mobile, edge, real-time systems.
---
## Core Concepts & Theory
### Teacher-Student Framework
Pretrained teacher guides student training.
### Temperature Scaling
Soften probability distributions; knowledge transfer.
### Feature Distillation
Intermediate feature alignment.
---
## Mathematical Formulation
Knowledge distillation loss:
$$L_{ ext{KD}} = \alpha L_{ ext{CE}}(y_s, y) + (1-\alpha) L_{ ext{CE}}(p_s/T, p_t/T)$$
where y_s = student logits, p_t = teacher soft targets, T = temperature.
---
## Advanced Theory & Extensions
### Attention Transfer
Spatial attention map alignment.
### FitNet
Feature map fitting; intermediate layers.
### Dark Knowledge
Non-obvious knowledge; probability distributions.
---
## Computational Considerations
Distillation overhead: O(teacher_forward).
Lightweight networks: Reduced parameters; faster inference.
Quantization: Further compression; integer arithmetic.
---
## Practical Implementation Strategies
### Temperature Tuning
3-20 typical; balance hardness.
### Loss Weighting
Balance cross-entropy and KD loss.
### Student Capacity
Cap student size; target deployment.
---
## Benchmark Datasets & Evaluation
CIFAR-10: KD standard benchmark.
ImageNet: Large-scale distillation.
Mobile: Deployment efficiency measure.
---
## Key Challenges & Limitations
### Student Capacity
Too small → cannot benefit.
### Teacher Quality
Better teacher → better student.
### Domain Shift
Distillation less effective across domains.
---
## Hyperparameter Tuning
Temperature T: 3-20; higher for softer.
Distillation weight α: 0.1-0.9; balance.
Student architecture: MobileNet, SqueezeNet.
---
## Real-World Applications & Case Studies
Mobile TensorFlow: Distilled models.
Edge Deployment: Lightweight NNs.
Production Systems: Latency-accuracy tradeoff.
---
## Integration with Other Methods
Distillation + Quantization → ultra-light.
Distillation + Pruning → multi-compression.
---
## Summary & Key Takeaways
Knowledge distillation enables model compression through teacher-student training with probability distribution alignment for efficient deployment.
Principles:
1. Teacher: large accurate model.
2. Student: small efficient model.
3. Temperature: probability softening.
4. KD loss: soft target supervision.
5. Multi-compression: combine techniques.
---
---
## Appendix: Practical Labs
### Lab 1: Knowledge Distillation Loss
import numpy as np
def kd_loss(teacher_logits, student_logits, true_labels, temperature=4.0, alpha=0.7):
"""Knowledge distillation loss"""
# Hard target loss (cross-entropy)
exp_student = np.exp(student_logits - np.max(student_logits, axis=1, keepdims=True))
probs_student = exp_student / exp_student.sum(axis=1, keepdims=True)
batch_size = len(true_labels)
hard_loss = -np.log(probs_student[np.arange(batch_size), true_labels] + 1e-8).mean()
# Soft target loss (distillation)
soft_teacher = np.exp(teacher_logits / temperature - np.max(teacher_logits, axis=1, keepdims=True) / temperature)
soft_teacher = soft_teacher / soft_teacher.sum(axis=1, keepdims=True)
soft_student = np.exp(student_logits / temperature - np.max(student_logits, axis=1, keepdims=True) / temperature)
soft_student = soft_student / soft_student.sum(axis=1, keepdims=True)
kl_loss = -np.sum(soft_teacher * np.log(soft_student + 1e-8), axis=1).mean()
# Combined loss
total_loss = alpha * hard_loss + (1 - alpha) * kl_loss * (temperature ** 2)
return total_loss
# Test
np.random.seed(42)
teacher_logits = np.random.randn(32, 10)
student_logits = np.random.randn(32, 10)
true_labels = np.random.randint(0, 10, 32)
loss = kd_loss(teacher_logits, student_logits, true_labels)
assert np.isfinite(loss), "Loss finite"
print("✓ KD loss working")
if __name__ == "__main__":
print("Lab 1: KDLoss - PASSED")### Lab 2: Temperature Effect
import numpy as np
def analyze_temperature_effect(logits, temperature_values=[1, 4, 10]):
"""Analyze temperature effect on softmax"""
results = {}
for T in temperature_values:
# Scale logits
scaled = logits / T
# Softmax
exp_scaled = np.exp(scaled - np.max(scaled, axis=1, keepdims=True))
probs = exp_scaled / exp_scaled.sum(axis=1, keepdims=True)
# Entropy
entropy = -np.sum(probs * np.log(probs + 1e-8), axis=1).mean()
# Max probability
max_prob = probs.max(axis=1).mean()
results[f"T_{T}"] = {
"entropy": entropy,
"max_prob": max_prob
}
return results
# Test
np.random.seed(42)
logits = np.random.randn(100, 10)
analysis = analyze_temperature_effect(logits)
assert len(analysis) == 3, "Three temperatures"
print("✓ Temperature analysis working")
if __name__ == "__main__":
print("Lab 2: TempAnalysis - PASSED")### Lab 3: Lightweight Architectures
import numpy as np
def compare_model_sizes(architectures):
"""Compare model complexities"""
results = {}
for name, arch in architectures.items():
# Count parameters
total_params = sum(layer["params"] for layer in arch)
# Compute latency (simplified)
latency = sum(layer["params"] / 1e6 for layer in arch) # Rough estimate
results[name] = {
"params": total_params,
"latency_ms": latency
}
return results
# Test
np.random.seed(42)
architectures = {
"ResNet50": [{"params": 25e6}],
"MobileNetV2": [{"params": 3.5e6}],
"SqueezeNet": [{"params": 1.2e6}]
}
comparison = compare_model_sizes(architectures)
assert "ResNet50" in comparison, "ResNet50 included"
assert comparison["MobileNetV2"]["params"] < comparison["ResNet50"]["params"], "Mobile smaller"
print("✓ Model size comparison working")
if __name__ == "__main__":
print("Lab 3: ModelComparison - PASSED")### Lab 4: Feature Distillation
import numpy as np
def feature_distillation_loss(teacher_features, student_features):
"""Align intermediate features"""
# Normalize features
teacher_norm = teacher_features / (np.linalg.norm(teacher_features, axis=1, keepdims=True) + 1e-8)
student_norm = student_features / (np.linalg.norm(student_features, axis=1, keepdims=True) + 1e-8)
# L2 loss
loss = np.mean((teacher_norm - student_norm) ** 2)
return loss
# Test
np.random.seed(42)
teacher_feat = np.random.randn(32, 256)
student_feat = np.random.randn(32, 256)
loss = feature_distillation_loss(teacher_feat, student_feat)
assert np.isfinite(loss), "Loss finite"
print("✓ Feature distillation working")
if __name__ == "__main__":
print("Lab 4: FeatureDistill - PASSED")