self distillation
**Self-distillation** trains a **model to match its own predictions on augmented or different views of data** — using the model itself as both teacher and student to improve consistency, regularization, and representation quality without requiring a separate larger model.
**What Is Self-Distillation?**
- **Definition**: Model learns from its own predictions.
- **Mechanism**: Match predictions across augmentations or training stages.
- **Goal**: Improve consistency and generalization.
- **Advantage**: No separate teacher model needed.
**Why Self-Distillation Works**
- **Consistency Regularization**: Same input should give same output.
- **Dark Knowledge**: Soft predictions contain useful structure.
- **Ensemble Effect**: Different views create implicit ensemble.
- **Denoising**: Averaged predictions reduce noise.
**Types of Self-Distillation**
**Temporal Self-Distillation** (Born-Again Networks):
```
1. Train model to convergence
2. Use final model as teacher
3. Train new model (same architecture) to match it
4. Repeat: often improves each generation
Model_1 → teaches → Model_2 → teaches → Model_3
(often better than Model_1)
```
**Layer-wise Self-Distillation**:
```svg
```
**Augmentation-Based**:
```
Original image → Prediction A
Augmented image → Prediction B
Loss: Match A and B (both from same model)
```
**Implementation**
**Augmentation Consistency**:
```python
import torch
import torch.nn.functional as F
def self_distillation_loss(model, x, augment_fn, temperature=4.0):
# Original prediction (teacher signal)
with torch.no_grad():
teacher_logits = model(x)
teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)
# Augmented prediction (student signal)
x_aug = augment_fn(x)
student_logits = model(x_aug)
student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)
# Consistency loss
consistency_loss = F.kl_div(
student_log_probs,
teacher_probs,
reduction="batchmean"
) * (temperature ** 2)
return consistency_loss
```
**Born-Again Training**:
```python
def born_again_training(model_class, dataset, generations=3):
"""Train successive generations of self-distillation."""
# Initial training
current_model = model_class()
train_standard(current_model, dataset)
for gen in range(generations - 1):
# Current model becomes teacher
teacher = current_model.eval()
# New student (same architecture)
student = model_class()
# Train student to match teacher
for x, y in dataset:
with torch.no_grad():
teacher_logits = teacher(x)
student_logits = student(x)
# Combine task loss and distillation loss
task_loss = F.cross_entropy(student_logits, y)
distill_loss = kl_divergence(student_logits, teacher_logits)
loss = 0.5 * task_loss + 0.5 * distill_loss
loss.backward()
optimizer.step()
current_model = student
print(f"Generation {gen + 1} complete")
return current_model
```
**Deep Layer Self-Distillation**:
```python
class SelfDistillationModel(nn.Module):
def __init__(self, base_model, num_classes):
super().__init__()
self.backbone = base_model
# Auxiliary classifiers at intermediate layers
self.aux_classifiers = nn.ModuleList([
nn.Linear(hidden_dim, num_classes)
for hidden_dim in intermediate_dims
])
self.final_classifier = nn.Linear(final_dim, num_classes)
def forward(self, x):
# Get intermediate features
features = self.backbone.get_intermediate_features(x)
# Auxiliary predictions
aux_logits = [clf(feat) for clf, feat in
zip(self.aux_classifiers, features[:-1])]
# Final prediction
final_logits = self.final_classifier(features[-1])
return final_logits, aux_logits
def compute_loss(self, x, labels):
final_logits, aux_logits = self.forward(x)
# Task loss
task_loss = F.cross_entropy(final_logits, labels)
# Self-distillation: intermediate layers match final
soft_targets = F.softmax(final_logits.detach() / 4.0, dim=-1)
distill_loss = sum(
F.kl_div(F.log_softmax(aux / 4.0, dim=-1), soft_targets)
for aux in aux_logits
)
return task_loss + 0.3 * distill_loss
```
**Applications**
**DINO (Self-Supervised Vision)**:
```
- Student and teacher share weights (EMA update)
- Different crops → should give same representation
- Learns powerful visual representations without labels
```
**Language Models**:
```
- Predict same output for paraphrased inputs
- Match representations of semantically similar text
- Improve robustness to input variations
```
**Benefits vs. Standard K.D.**
```
Aspect | Self-Distillation | Teacher-Student
--------------------|--------------------|-----------------
Teacher required | No | Yes
Architecture | Same | Different allowed
Training simplicity | Higher | Lower
Max performance | Good | Better (bigger teacher)
Use case | Regularization | Compression
```
Self-distillation is **a powerful regularization technique** — by forcing models to be consistent across views or to match their own refined predictions, it improves generalization without the complexity of maintaining separate teacher models.