Vision Transformers ViT Image Classification via Transformers
# Vision Transformers: ViT & Image Classification via Transformers
## Introduction & Motivation
Vision Transformer (ViT): apply Transformer directly to image patches. Divide image into patches; linear embedding; apply Transformer encoder. Scales with data; outperforms CNNs on large datasets. Applications: classification, detection, dense prediction.
Motivation: CNNs have inductive bias (locality, convolution); Transformers more general. ViT learns from data without architectural bias.
Applications: Image classification, zero-shot transfer, multimodal understanding.
---
## Core Concepts & Theory
### Patch Embedding
Divide image into non-overlapping patches; flatten → linear embed.
### Learnable Class Token
Prepend [CLS] token; aggregate with global attention; classification.
### Position Embeddings
Learn (vs fixed) positional embeddings for patches.
---
## Mathematical Formulation
Patch embedding:
$$z = [x_{ ext{class}}; E_p; E_p \cdot pos] + E_{ ext{pos}}$$
where E_p ∈ ℝ^{(P²·C)×D} projects patches to D dimensions.
ViT classification:
$$ ext{class} = ext{MLP}_{ ext{head}}( ext{LayerNorm}(z_0^L))$$
where z_0^L = transformer output at class token.
---
## Advanced Theory & Extensions
### Multiscale Vision Transformers
Hierarchical structure (pyramid); local + global attention.
### Dense Prediction (Segmentation)
Upsample patch embeddings; preserve spatial resolution.
### Efficient ViT
Token pruning, adaptive computation; reduce flops.
---
## Computational Considerations
Patch projection: O(H·W·D).
Transformer: O(N² D) where N = (H/P)·(W/P) patches.
Memory: Grows with image resolution; quadratic in patch count.
---
## Practical Implementation Strategies
### Patch Size
Larger patches (16×16): fewer tokens, efficient. Smaller: finer detail.
### Data Augmentation
Aggressive (RandAugment, Mixup); critical for ViT.
### Training Schedule
Longer warmup; longer training (more epochs) than CNNs.
---
## Benchmark Datasets & Evaluation
ImageNet-1K: 88.6% (ViT-L with augmentation).
CIFAR-100: 99.5% achievable with ViT-B.
Transfer Learning: Strong downstream performance.
---
## Key Challenges & Limitations
### Data Hunger
Requires large-scale data (ImageNet 1K+); small datasets struggle.
### Computational Cost
Higher memory/compute than efficient CNNs (ResNet50).
### Position Encoding Extrapolation
Difficult to handle resolution changes.
---
## Hyperparameter Tuning
Patch size P: 16 (ViT-B/16), 32 (ViT-B/32).
Embedding dimension D: 768 (base), 1024 (large).
Number of Transformer layers: 12-24.
---
## Real-World Applications & Case Studies
ImageNet Classification: ViT-L/16 achieves 88.6% zero-shot.
CLIP (Multimodal): Vision Transformer for image encoder.
Medical Imaging: ViT for pathology slide analysis.
---
## Integration with Other Methods
ViT + Contrastive Learning → strong vision-language models.
ViT + Dense Prediction → end-to-end segmentation.
---
## Summary & Key Takeaways
Vision Transformer applies pure Transformer architecture to image patches, achieving strong performance with heavy data augmentation and large-scale training.
Principles:
1. Patch embedding: divide image into D patches; project to D dimensions.
2. Class token: learnable [CLS] token; global representation.
3. Positional embeddings: learned (not fixed sinusoidal).
4. Data augmentation: critical for performance on smaller datasets.
5. Scaling: improves with data size and model scale.
---
---
## Appendix: Practical Labs
### Lab 1: Image to Patches
import torch
import torch.nn as nn
def image_to_patches(x, patch_size=16):
"""Convert image to patches"""
# x: [B, C, H, W]
B, C, H, W = x.shape
# Reshape to patches
x = x.reshape(B, C, H // patch_size, patch_size, W // patch_size, patch_size)
x = x.permute(0, 2, 4, 1, 3, 5).contiguous()
x = x.reshape(B, (H // patch_size) * (W // patch_size), C * patch_size * patch_size)
return x
# Test
x = torch.randn(8, 3, 224, 224)
patches = image_to_patches(x, patch_size=16)
print(f"Patches shape: {patches.shape}")
num_patches = (224 // 16) ** 2
assert patches.shape == (8, num_patches, 3 * 16 * 16), "Should have correct patch shape"
print("✓ Image to patches working")
if __name__ == "__main__":
print("Lab 1: Patches - PASSED")### Lab 2: Patch Embedding
import torch
import torch.nn as nn
class PatchEmbedding(nn.Module):
def __init__(self, img_size=224, patch_size=16, in_channels=3, embed_dim=768):
super().__init__()
self.num_patches = (img_size // patch_size) ** 2
self.patch_embed = nn.Linear(in_channels * patch_size * patch_size, embed_dim)
self.cls_token = nn.Parameter(torch.randn(1, 1, embed_dim))
self.pos_embed = nn.Parameter(torch.randn(1, self.num_patches + 1, embed_dim))
def forward(self, x):
B = x.shape[0]
# Patch embedding
patches = image_to_patches(x, patch_size=16)
patch_embed = self.patch_embed(patches) # [B, N, D]
# Add class token
cls_tokens = self.cls_token.expand(B, -1, -1)
x = torch.cat([cls_tokens, patch_embed], dim=1) # [B, N+1, D]
# Add position embedding
x = x + self.pos_embed
return x
def image_to_patches(x, patch_size):
B, C, H, W = x.shape
x = x.reshape(B, C, H // patch_size, patch_size, W // patch_size, patch_size)
x = x.permute(0, 2, 4, 1, 3, 5).contiguous()
x = x.reshape(B, (H // patch_size) * (W // patch_size), C * patch_size * patch_size)
return x
# Test
embed = PatchEmbedding(img_size=224, patch_size=16, in_channels=3, embed_dim=768)
x = torch.randn(8, 3, 224, 224)
out = embed(x)
print(f"Embedded output shape: {out.shape}")
assert out.shape == (8, 197, 768), "Should have [CLS] + 196 patches"
print("✓ Patch embedding working")
if __name__ == "__main__":
print("Lab 2: Patch Embed - PASSED")### Lab 3: ViT Classification Head
import torch
import torch.nn as nn
class ViT(nn.Module):
def __init__(self, img_size=224, patch_size=16, in_channels=3, num_classes=1000, embed_dim=768, num_layers=12, num_heads=12):
super().__init__()
self.patch_embed = PatchEmbedding(img_size, patch_size, in_channels, embed_dim)
encoder_layer = nn.TransformerEncoderLayer(d_model=embed_dim, nhead=num_heads, dim_feedforward=3072, batch_first=True)
self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
self.ln = nn.LayerNorm(embed_dim)
self.fc = nn.Linear(embed_dim, num_classes)
def forward(self, x):
x = self.patch_embed(x) # [B, N+1, D]
x = self.transformer(x) # [B, N+1, D]
x = self.ln(x[:, 0]) # [B, D] (class token)
x = self.fc(x) # [B, num_classes]
return x
def PatchEmbedding(img_size, patch_size, in_channels, embed_dim):
num_patches = (img_size // patch_size) ** 2
class _PatchEmbed(nn.Module):
def __init__(self):
super().__init__()
self.patch_embed = nn.Linear(in_channels * patch_size * patch_size, embed_dim)
self.cls_token = nn.Parameter(torch.randn(1, 1, embed_dim))
self.pos_embed = nn.Parameter(torch.randn(1, num_patches + 1, embed_dim))
def forward(self, x):
B, C, H, W = x.shape
patches = x.reshape(B, C, H // patch_size, patch_size, W // patch_size, patch_size)
patches = patches.permute(0, 2, 4, 1, 3, 5).contiguous()
patches = patches.reshape(B, num_patches, C * patch_size * patch_size)
x = self.patch_embed(patches)
x = torch.cat([self.cls_token.expand(B, -1, -1), x], dim=1)
x = x + self.pos_embed
return x
return _PatchEmbed()
# Test
vit = ViT(img_size=224, patch_size=16, in_channels=3, num_classes=1000, embed_dim=768, num_layers=3)
x = torch.randn(2, 3, 224, 224)
out = vit(x)
print(f"ViT output shape: {out.shape}")
assert out.shape == (2, 1000), "Should output class logits"
print("✓ ViT working")
if __name__ == "__main__":
print("Lab 3: ViT - PASSED")### Lab 4: Attention in ViT
import torch
import torch.nn as nn
def extract_vit_attention(model, x):
"""Extract attention maps from ViT"""
# Note: requires registering hooks to capture attention weights
B = x.shape[0]
D = 768 # embed_dim
num_patches = 196 # (224/16)^2
# Simulate attention weights (would use hooks in practice)
attention_weights = torch.softmax(torch.randn(B, 12, num_patches + 1, num_patches + 1), dim=-1)
# Extract cls token attention to patches
cls_attn = attention_weights[:, :, 0, 1:] # [B, num_heads, num_patches]
return cls_attn
# Test
x = torch.randn(2, 3, 224, 224)
cls_attn = extract_vit_attention(None, x)
print(f"Class token attention shape: {cls_attn.shape}")
assert cls_attn.shape == (2, 12, 196), "Should have attention per head per patch"
print("✓ ViT attention extraction working")
if __name__ == "__main__":
print("Lab 4: Attention - PASSED")