Vision Transformer - ViT
# Vision Transformer - ViT
## Introduction & Motivation
Vision Transformer: apply transformer architecture to image classification. Patch embeddings, self-attention for spatial features. Applications: image classification, transfer learning.
Motivation: Scale vision models with transformer architecture.
Applications: Image classification, transfer learning foundation.
---
## Core Concepts & Theory
### Image Patches
Divide image into patches.
### Patch Embeddings
Linear projection of patches.
### Positional Encoding
Add spatial position information.
### Class Token
Aggregate information for classification.
---
## Mathematical Formulation
Patch Embedding:
$$E = ext{Linear}( ext{Flatten}(P)) \in \mathbb{R}^{N imes D}$$
Positional Encoding:
$$X = [x_{ ext{cls}}, E + ext{pos}] \in \mathbb{R}^{(N+1) imes D}$$
Transformer:
$$ ext{TransformerBlock}(X) = ext{MLP}( ext{LayerNorm}( ext{MSA}( ext{LayerNorm}(X)) + X)) + X$$
---
## Advanced Theory & Extensions
### Hierarchical ViT
Multi-scale patch embedding.
### Hybrid Models
CNN + Transformer.
### Efficient Variants
Reduce computational cost.
---
## Computational Considerations
Patch embedding: O(N·D²).
Self-attention: O(N²·D).
Forward pass: O(N·D²).
---
## Practical Implementation Strategies
### Patch Size Selection
Balance spatial and temporal dimensions.
### Positional Encoding
Learnable or fixed sinusoidal.
### Resolution Scaling
Interpolate for different resolutions.
---
## Benchmark Datasets & Evaluation
ImageNet: Image classification.
CIFAR: Small-scale images.
ImageNet-21K: Large-scale pre-training.
---
## Key Challenges & Limitations
### Data Requirements
Needs large-scale pre-training.
### Computational Cost
Quadratic attention complexity.
### Inductive Bias
Limited spatial priors.
---
## Hyperparameter Tuning
Patch size: 4, 8, 16.
Embedding dimension: 256-1024.
Number of heads: 8-16.
---
## Real-World Applications & Case Studies
Image Classification: Strong transfer learning.
Medical Imaging: Diagnosis tasks.
Fine-Grained Recognition: Detailed classification.
---
## Integration with Other Methods
ViT + knowledge distillation; + multi-scale features for efficiency.
---
## Summary & Key Takeaways
Vision Transformer adapts transformer to vision effectively.
Principles:
1. Patch embedding: Divide into patches.
2. Self-attention: Global context.
3. Position encoding: Spatial information.
4. Scalability: Scales with data.
5. Transfer learning: Strong pre-training.
---
## Appendix: Practical Labs
### Lab 1: Patch Embedding
import numpy as np
def patch_embedding(image, patch_size=16):
"""Extract patch embeddings from image"""
h, w, c = image.shape
patches = []
for i in range(0, h, patch_size):
for j in range(0, w, patch_size):
patch = image[i:i+patch_size, j:j+patch_size]
flat_patch = patch.flatten()
patches.append(flat_patch)
return np.array(patches)
np.random.seed(42)
img = np.random.rand(224, 224, 3)
patches = patch_embedding(img, patch_size=16)
assert patches.shape[0] == (224//16) ** 2
print(f"✓ Patch embedding: {patches.shape}")### Lab 2: Positional Encoding
import numpy as np
def positional_encoding(seq_len, d_model):
"""Create positional encoding"""
pos = np.arange(seq_len)[:, np.newaxis]
div = np.exp(np.arange(0, d_model, 2) * -(np.log(10000) / d_model))
pos_enc = np.zeros((seq_len, d_model))
pos_enc[:, 0::2] = np.sin(pos * div)
pos_enc[:, 1::2] = np.cos(pos * div)
return pos_enc
pos_enc = positional_encoding(196 + 1, 768)
assert pos_enc.shape == (197, 768)
print("✓ Positional encoding working")### Lab 3: Class Token
import numpy as np
def add_class_token(patch_embeddings):
"""Add learnable class token to patches"""
batch_size, num_patches, d_model = patch_embeddings.shape
cls_token = np.random.randn(batch_size, 1, d_model) * 0.01
embeddings_with_cls = np.concatenate([cls_token, patch_embeddings], axis=1)
return embeddings_with_cls
np.random.seed(42)
patches = np.random.randn(4, 196, 768)
with_cls = add_class_token(patches)
assert with_cls.shape == (4, 197, 768)
print("✓ Class token addition working")### Lab 4: Multi-Head Attention on Patches
import numpy as np
def multi_head_attention_vision(query, key, value, num_heads=8):
"""Multi-head attention for vision patches"""
batch_size, num_patches, d_model = query.shape
head_dim = d_model // num_heads
# Reshape for multiple heads
query = query.reshape(batch_size, num_patches, num_heads, head_dim).transpose(0, 2, 1, 3)
key = key.reshape(batch_size, num_patches, num_heads, head_dim).transpose(0, 2, 1, 3)
value = value.reshape(batch_size, num_patches, num_heads, head_dim).transpose(0, 2, 1, 3)
# Attention
scores = query @ key.transpose(0, 1, 3, 2) / np.sqrt(head_dim)
attention = np.exp(scores) / np.sum(np.exp(scores), axis=-1, keepdims=True)
# Apply to values
output = attention @ value
# Reshape back
output = output.transpose(0, 2, 1, 3).reshape(batch_size, num_patches, d_model)
return output
np.random.seed(42)
q = np.random.randn(2, 197, 768)
k = np.random.randn(2, 197, 768)
v = np.random.randn(2, 197, 768)
out = multi_head_attention_vision(q, k, v)
assert out.shape == (2, 197, 768)
print("✓ Multi-head attention working")---