Distilbert - Knowledge Distillation

# DistilBERT - Knowledge Distillation

## Introduction & Motivation

DistilBERT: distilled BERT via knowledge distillation. 40% smaller, 60% faster than BERT. Applications: mobile, real-time inference, resource-constrained deployment.

Motivation: Compress BERT for efficient deployment.

Applications: Edge inference, mobile NLP, low-latency systems.

---

## Core Concepts & Theory

### Knowledge Distillation

Transfer knowledge from teacher to student.

### Attention Distillation

Distill attention patterns.

### Temperature Scaling

Soften probability distributions.

### Layer Reduction

Fewer transformer layers.

---

## Mathematical Formulation

KL Divergence Loss:
$$\mathcal{L}_{ ext{KL}} = T^2 \cdot ext{KL}(p_{ ext{teacher}}, p_{ ext{student}})$$

Distillation Loss:
$$\mathcal{L}_{ ext{dist}} = \alpha \cdot \mathcal{L}_{ ext{CE}} + (1-\alpha) \cdot \mathcal{L}_{ ext{KL}}$$

Temperature:
$$p_i = \frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}$$

---

## Advanced Theory & Extensions

### Attention Transfer

Transfer attention head weights.

### Feature-Based Distillation

Distill intermediate representations.

### Multi-Head Distillation

Collapse multiple heads.

---

## Computational Considerations

Teacher inference: O(T·H²).

Student inference: O(T'·H'²) where T' < T, H' < H.

Speedup: ~2x faster, 40% fewer parameters.

---

## Practical Implementation Strategies

### Temperature Selection

Typical range 3-20.

### Loss Weighting

Balance CE and KL losses.

### Progressive Distillation

Layer-by-layer distillation.

---

## Benchmark Datasets & Evaluation

GLUE: Text understanding.

SQuAD: Reading comprehension.

Inference latency: Deployment benchmarks.

---

## Key Challenges & Limitations

### Knowledge Bottleneck

Limited information transfer.

### Task-Specific Distillation

Different optimal temperatures per task.

### Hyperparameter Sensitivity

Fine-tuning required.

---

## Hyperparameter Tuning

Temperature: 3-20.

Alpha (loss weight): 0.1-0.5.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Mobile Apps: On-device inference.

Browser ML: Client-side NLP.

IoT Devices: Embedded NLP.

---

## Integration with Other Methods

DistilBERT + quantization; + pruning for further compression.

---

## Summary & Key Takeaways

DistilBERT efficiently compresses BERT via knowledge distillation.

Principles:
1. Knowledge distillation: Transfer from teacher.
2. Layer reduction: Fewer transformer blocks.
3. Temperature scaling: Soften distributions.
4. Attention transfer: Compress attention patterns.
5. Efficiency: 2x speedup, 40% smaller.

---

## Appendix: Practical Labs

### Lab 1: KL Divergence Loss

import numpy as np

def kl_divergence_loss(teacher_logits, student_logits, temperature=3.0):
 """Compute KL divergence between teacher and student"""
 teacher_probs = np.exp(teacher_logits / temperature) / np.sum(np.exp(teacher_logits / temperature))
 student_probs = np.exp(student_logits / temperature) / np.sum(np.exp(student_logits / temperature))
 
 kl = np.sum(teacher_probs * (np.log(teacher_probs + 1e-8) - np.log(student_probs + 1e-8)))
 return kl

np.random.seed(42)
teacher = np.random.randn(10)
student = np.random.randn(10)
loss = kl_divergence_loss(teacher, student)
assert loss > 0
print(f"✓ KL loss: {loss:.4f}")

### Lab 2: Temperature Scaling

import numpy as np

def temperature_scaling(logits, temperature=1.0):
 """Apply temperature to logits"""
 scaled = logits / temperature
 probs = np.exp(scaled) / np.sum(np.exp(scaled))
 return probs

np.random.seed(42)
logits = np.random.randn(10)
probs_hard = temperature_scaling(logits, T=1.0)
probs_soft = temperature_scaling(logits, T=5.0)
assert np.max(probs_soft) < np.max(probs_hard)
print("✓ Temperature scaling working")

### Lab 3: Distillation Loss

import numpy as np

def distillation_loss(ce_loss, kl_loss, alpha=0.3, temperature=3.0):
 """Combine CE and KL losses"""
 total = alpha * ce_loss + (1 - alpha) * (temperature ** 2) * kl_loss
 return total

ce = 0.8
kl = 0.3
loss = distillation_loss(ce, kl, alpha=0.3)
assert loss > 0
print(f"✓ Distillation loss: {loss:.3f}")

### Lab 4: Attention Transfer

import numpy as np

def attention_transfer_loss(teacher_attn, student_attn):
 """Transfer attention patterns from teacher to student"""
 # MSE loss on attention weights
 loss = np.mean((teacher_attn - student_attn) ** 2)
 return loss

np.random.seed(42)
teacher = np.random.rand(10, 10)
student = np.random.rand(10, 10)
loss = attention_transfer_loss(teacher, student)
assert loss > 0
print(f"✓ Attention transfer loss: {loss:.4f}")

---