Deit - Data-Efficient Image Transformers

# DeiT - Data-Efficient Image Transformers

## Introduction & Motivation

DeiT: data-efficient vision transformers via knowledge distillation. Train on ImageNet without large-scale pre-training. Applications: practical vision transformers, transfer learning.

Motivation: Make ViT practical without ImageNet-21K pre-training.

Applications: Efficient vision models, practical deployment.

---

## Core Concepts & Theory

### Knowledge Distillation

Transfer from CNN teacher.

### Attention Distillation

Distill attention patterns.

### Data Augmentation

RandAugment, CutMix for efficiency.

### Efficient Training

Optimized training procedure.

---

## Mathematical Formulation

Distillation Loss:
$$\mathcal{L} = \alpha \mathcal{L}_{ ext{CE}} + (1-\alpha) \mathcal{L}_{ ext{KL}}$$

Attention Distillation:
$$\mathcal{L}_{ ext{attn}} = ext{MSE}( ext{Attn}_{ ext{teacher}}, ext{Attn}_{ ext{student}})$$

KL Divergence:
$$ ext{KL}(p || q) = \sum_i p_i \log(p_i / q_i)$$

---

## Advanced Theory & Extensions

### Token Distillation

Distill intermediate representations.

### Hard Negative Mining

Focus on challenging examples.

### Multi-Crop Training

Different augmentation strategies.

---

## Computational Considerations

Teacher inference: O(H·W).

Student inference: O(N²·D).

Distillation overhead: Minimal.

---

## Practical Implementation Strategies

### Teacher Selection

Use strong CNN baseline.

### Temperature Tuning

Balance soft and hard targets.

### Augmentation Strategy

CutMix, Mixup for regularization.

---

## Benchmark Datasets & Evaluation

ImageNet: Classification benchmark.

ImageNet-V2: Out-of-distribution evaluation.

Fine-tuning tasks: Downstream performance.

---

## Key Challenges & Limitations

### Teacher Quality

Limited by teacher performance.

### Computational Cost

Still requires GPU resources.

### Hyperparameter Tuning

Sensitive to training details.

---

## Hyperparameter Tuning

Temperature: 1.0-5.0.

Alpha (loss weight): 0.3-0.7.

Learning rate: 1e-4 to 1e-3.

---

## Real-World Applications & Case Studies

Mobile Deployment: Lightweight ViT variants.

Fine-Grained Classification: Transfer to specialized tasks.

Medical Imaging: Domain-specific applications.

---

## Integration with Other Methods

DeiT + quantization for further compression; + pruning for efficiency.

---

## Summary & Key Takeaways

DeiT enables practical vision transformers through distillation.

Principles:
1. Knowledge distillation: Transfer from CNN.
2. Attention distillation: Transfer attention patterns.
3. Data augmentation: RandAugment, CutMix.
4. Efficient training: Optimized procedures.
5. Practical: ImageNet training sufficient.

---

## Appendix: Practical Labs

### Lab 1: Token Distillation

import numpy as np

def token_distillation(student_tokens, teacher_tokens, alpha=0.5):
 """Distill token representations"""
 # MSE loss between token embeddings
 token_loss = np.mean((student_tokens - teacher_tokens) ** 2)
 return token_loss

np.random.seed(42)
student = np.random.randn(4, 197, 768)
teacher = np.random.randn(4, 197, 768)
loss = token_distillation(student, teacher)
assert loss > 0
print(f"✓ Token distillation loss: {loss:.3f}")

### Lab 2: Attention Distillation

import numpy as np

def attention_distillation(student_attn, teacher_attn):
 """Distill attention head patterns"""
 # MSE between attention matrices
 loss = np.mean((student_attn - teacher_attn) ** 2)
 return loss

np.random.seed(42)
student_a = np.random.rand(12, 197, 197)
teacher_a = np.random.rand(12, 197, 197)
loss = attention_distillation(student_a, teacher_a)
assert loss > 0
print(f"✓ Attention distillation loss: {loss:.4f}")

### Lab 3: CutMix Augmentation

import numpy as np

def cutmix(image1, image2, alpha=1.0):
 """CutMix data augmentation"""
 lam = np.random.beta(alpha, alpha)
 
 h, w = image1.shape[:2]
 cut_ratio = np.sqrt(1 - lam)
 cut_h = int(h * cut_ratio)
 cut_w = int(w * cut_ratio)
 
 cx = np.random.randint(0, w)
 cy = np.random.randint(0, h)
 
 x1 = np.clip(cx - cut_w // 2, 0, w)
 y1 = np.clip(cy - cut_h // 2, 0, h)
 x2 = np.clip(cx + cut_w // 2, 0, w)
 y2 = np.clip(cy + cut_h // 2, 0, h)
 
 mixed = image1.copy()
 mixed[y1:y2, x1:x2] = image2[y1:y2, x1:x2]
 
 return mixed, lam

np.random.seed(42)
img1 = np.random.rand(224, 224, 3)
img2 = np.random.rand(224, 224, 3)
mixed, lam = cutmix(img1, img2)
assert mixed.shape == img1.shape
print(f"✓ CutMix: lambda={lam:.3f}")

### Lab 4: RandAugment

import numpy as np

def randaugment(image, num_ops=2, magnitude=9):
 """Apply random augmentation operations"""
 ops = ['rotation', 'shear', 'brightness', 'contrast']
 
 for _ in range(num_ops):
 op = np.random.choice(ops)
 level = np.random.randint(0, magnitude)
 
 if op == 'rotation':
 angle = (level / magnitude) * 30
 # Rotation logic
 elif op == 'brightness':
 factor = 1 + (level / magnitude) * 0.3
 image = np.clip(image * factor, 0, 1)
 # Other operations...
 
 return image

np.random.seed(42)
img = np.random.rand(224, 224, 3)
aug = randaugment(img)
assert aug.shape == img.shape
print("✓ RandAugment working")

---