Knowledge Distillation for Model Compression

# Knowledge Distillation for Model Compression

## Introduction & Motivation

Knowledge distillation transfers learning from large models to small ones, enabling deployment on resource-constrained devices. Critical for edge computing, embedded systems, and real-time applications in manufacturing and autonomous systems.

Motivation: Compress models while maintaining performance.

Applications: Model compression, edge deployment, real-time inference, resource-constrained systems.

---

## Core Concepts & Theory

### Teacher-Student Learning

Knowledge transfer mechanism.

### Dark Knowledge

Soft target distributions.

### Temperature Scaling

Softmax smoothing.

### Transfer Loss

KL divergence to teacher.

---

## Mathematical Formulation

Distillation Loss:
$$L = \alpha L_{CE}(y, \hat{y}) + (1-\alpha) L_{KL}(p_T, p_S)$$

Soft Targets:
$$p_i^{soft} = \frac{e^{z_i/T}}{\sum_j e^{z_j/T}}$$

KL Divergence:
$$D_{KL}(P||Q) = \sum_i P_i \log \frac{P_i}{Q_i}$$

---

## Advanced Theory & Extensions

### Attention Transfer

Feature map distillation.

### Feature-Based Distillation

Intermediate layer matching.

### Multi-Teacher Ensemble

Ensemble as teacher.

---

## Computational Considerations

Teacher: O(D_T) inference.

Student: O(D_S) inference.

Distillation: O(D_T + D_S) training.

---

## Practical Implementation Strategies

### Temperature Selection

Balancing hardness of targets.

### Loss Weight Tuning

Accuracy vs. distillation.

### Architecture Design

Capacity ratios.

---

## Benchmark Datasets & Evaluation

ImageNet: Vision tasks.

CIFAR: Small-scale classification.

Domain Models: Specialized tasks.

---

## Key Challenges & Limitations

### Architecture Mismatch

Very different teacher-student.

### Compression Limits

Minimum viable student size.

### Training Stability

Temperature effects.

---

## Hyperparameter Tuning

Temperature: 3-20.

Distillation weight: 0.1-0.9.

Learning rate: 1e-4 to 1e-2.

---

## Real-World Applications & Case Studies

Mobile Inference: On-device deployment.

Real-Time Systems: Edge processing.

IoT Devices: Resource constraints.

---

## Integration with Other Methods

Knowledge distillation + compression; + pruning; + quantization.

---

## Summary & Key Takeaways

Knowledge distillation enables efficient deployment.

Principles:
1. Teacher: Large, accurate model.
2. Student: Compact architecture.
3. Transfer: Dark knowledge sharing.
4. Temperature: Soft target adjustment.
5. Deployment: Resource-efficient inference.

---

## Appendix: Practical Labs

### Lab 1: Soft Target Generation

import numpy as np

def generate_soft_targets(teacher_logits, temperature=4.0):
 """Create soft targets from teacher"""
 scaled = teacher_logits / temperature
 exp_logits = np.exp(scaled)
 soft_targets = exp_logits / np.sum(exp_logits)
 return soft_targets

logits = np.array([2.0, 1.0, 0.5])
soft = generate_soft_targets(logits, temperature=4.0)

print(f"✓ Soft targets: {soft}")

### Lab 2: Distillation Loss

import numpy as np

def distillation_loss(student_logits, teacher_logits, labels, temperature=4.0, alpha=0.7):
 """Combined distillation and CE loss"""
 # KL divergence loss
 student_soft = np.exp(student_logits / temperature) / np.sum(np.exp(student_logits / temperature))
 teacher_soft = np.exp(teacher_logits / temperature) / np.sum(np.exp(teacher_logits / temperature))
 
 kl_loss = -np.sum(teacher_soft * np.log(student_soft + 1e-10))
 
 # Cross-entropy loss
 ce_loss = -np.sum(labels * np.log(np.exp(student_logits) / np.sum(np.exp(student_logits)) + 1e-10))
 
 total_loss = alpha * ce_loss + (1 - alpha) * kl_loss
 return total_loss

s_logits = np.array([2.0, 1.0, 0.5])
t_logits = np.array([2.5, 1.2, 0.3])
labels = np.array([1, 0, 0])

loss = distillation_loss(s_logits, t_logits, labels, temperature=4.0)
print(f"✓ Distillation loss: {loss:.4f}")

### Lab 3: Teacher-Student Training

import numpy as np

class KnowledgeDistillation:
 def __init__(self, teacher_weights, student_weights):
 self.teacher = teacher_weights
 self.student = student_weights
 
 def train_student(self, X, labels, n_epochs=50, alpha=0.7, T=4.0):
 """Train student to mimic teacher"""
 for epoch in range(n_epochs):
 # Teacher predictions
 teacher_out = X @ self.teacher
 
 # Student predictions
 student_out = X @ self.student
 
 # Distillation loss (simplified)
 temp_factor = 1.0 / T
 kl_loss = np.sum((teacher_out - student_out) ** 2) * temp_factor
 
 ce_loss = np.sum((student_out - labels) ** 2)
 
 total_loss = alpha * ce_loss + (1 - alpha) * kl_loss
 
 # Update student
 grad = 2 * (student_out - labels) @ X.T
 self.student -= 0.01 * grad

distill = KnowledgeDistillation(np.random.randn(10, 1), np.random.randn(10, 1))
X = np.random.randn(50, 10)
labels = X[:, 0:1] + np.random.randn(50, 1) * 0.1

distill.train_student(X, labels)
print(f"✓ Student training complete")

### Lab 4: Compression Evaluation

import numpy as np

def evaluate_compression(model_size_original, model_size_compressed, accuracy_original, accuracy_student):
 """Evaluate compression effectiveness"""
 compression_ratio = model_size_original / model_size_compressed
 accuracy_drop = accuracy_original - accuracy_student
 
 efficiency = compression_ratio / max(accuracy_drop, 0.01)
 
 return {
 'compression_ratio': compression_ratio,
 'accuracy_drop': accuracy_drop,
 'efficiency': efficiency
 }

metrics = evaluate_compression(
 model_size_original=100, # MB
 model_size_compressed=10, # MB
 accuracy_original=0.95,
 accuracy_student=0.92
)

print(f"✓ Compression: {metrics['compression_ratio']:.1f}x")
print(f"✓ Accuracy drop: {metrics['accuracy_drop']*100:.1f}%")

---