vision transformers vit patch embedding
# Vision Transformers: ViT & Patch Embedding
## Introduction & Motivation
Vision transformers: apply transformers to images. Patch embedding: divide image into patches; embed linearly. Classification token: learnable token for image representation. Positional encoding: 2D patch positions. Applications: image classification, object detection, image segmentation.
Motivation: CNNs rely on inductive biases; transformers data-driven. ViT achieves competitive performance; enables scaling.
Applications: Image classification, multimodal, vision-language.
---
## Core Concepts & Theory
### Patch Embedding
Image → fixed-size patches; linear projection.
### Classification Token (CLS)
Learnable token; output representation.
### 2D Positional Encoding
Encode patch positions; 1D or 2D learnable.
---
## Mathematical Formulation
Patch embedding:
$$x_p = ext{LinearProj}( ext{Flatten}( ext{Patch}))$$
where patch ∈ ℝ^{P×P×C}, output ∈ ℝ^D.
CLS token + patches:
$$x = [x_{ ext{cls}}, x_1, \ldots, x_N] + E_{ ext{pos}}$$
where N = number patches.
ViT forward:
$$y = ext{MLP}( ext{Transformer}(x))$$
---
## Advanced Theory & Extensions
### Hybrid Architectures
CNN backbone → transformer head.
### Multi-Scale Patches
Hierarchical patch sizes; HiT.
### Window Attention
Local attention windows; efficiency.
---
## Computational Considerations
Patch embedding: O(P² · C · D) per image.
Transformer: O(N² · D²) quadratic in number patches.
ViT-Base: ~86M parameters; competitive with ResNet-50.
---
## Practical Implementation Strategies
### Patch Overlap
Non-overlapping standard; overlapping possible.
### Positional Interpolation
Interpolate PE for different resolutions.
### Layer Scaling
Deeper ViT; 24+ layers typical.
---
## Benchmark Datasets & Evaluation
ImageNet: ViT competitive; requires pretraining.
ImageNet-21k: Pretraining dataset; large-scale.
COCO: Object detection; detection head addition.
---
## Key Challenges & Limitations
### Data Requirement
Needs large pretraining; not optimal for small datasets.
### Computational Cost
O(N²) quadratic; longer sequences expensive.
### Local Structure
Lacks local inductive bias; requires more data.
---
## Hyperparameter Tuning
Patch size: 16x16 standard; 14x14, 32x32 variants.
Number of layers: 12-24; depth scaling.
Hidden dimension: 768-1024; width scaling.
---
## Real-World Applications & Case Studies
ImageNet Classification: ViT-L pretrained; fine-tune efficient.
Object Detection: DETR; end-to-end detection.
Semantic Segmentation: SETR; transformer segmentation.
---
## Integration with Other Methods
ViT + CNN → hybrid; combine inductive bias + flexibility.
ViT + Self-Supervised → DINO, MAE.
---
## Summary & Key Takeaways
Vision transformers via patch embedding and transformer layers enable competitive image classification with scalable pretraining.
Principles:
1. Patch embedding: image → tokens.
2. CLS token: global representation.
3. Positional encoding: spatial awareness.
4. Scaling: large pretraining beneficial.
5. Fine-tuning: transfer learning efficiency.
---
---
## Appendix: Practical Labs
### Lab 1: Patch Embedding
import numpy as np
def patch_embedding(image, patch_size=16):
"""Extract and embed patches"""
H, W, C = image.shape
P = patch_size
# Extract patches
patches = []
for i in range(0, H, P):
for j in range(0, W, P):
patch = image[i:i+P, j:j+P, :]
if patch.shape[:2] == (P, P):
patches.append(patch.flatten())
patches = np.array(patches) # (num_patches, P*P*C)
# Linear projection
D = 768 # embedding dimension
W_proj = np.random.randn(patches.shape[1], D)
embeddings = np.dot(patches, W_proj) # (num_patches, D)
return embeddings
# Test
np.random.seed(42)
image = np.random.randn(224, 224, 3)
embeddings = patch_embedding(image, patch_size=16)
assert embeddings.shape[1] == 768, "Embedding dimension"
assert embeddings.shape[0] == (224 // 16) ** 2, "Number of patches"
print("✓ Patch embedding working")
if __name__ == "__main__":
print("Lab 1: PatchEmbedding - PASSED")### Lab 2: CLS Token & Positional Encoding
import numpy as np
def vit_sequence_construction(patches, D=768):
"""Add CLS token and positional encoding"""
num_patches = patches.shape[0]
# CLS token
cls_token = np.random.randn(1, D)
# Combine
sequence = np.vstack([cls_token, patches]) # (num_patches+1, D)
# Positional encoding (learnable or sinusoidal)
positions = np.arange(num_patches + 1)[:, np.newaxis]
div_term = np.exp(np.arange(0, D, 2) * -(np.log(10000.0) / D))
pos_enc = np.zeros((num_patches + 1, D))
pos_enc[:, 0::2] = np.sin(positions * div_term)
pos_enc[:, 1::2] = np.cos(positions * div_term)
# Add positional encoding
sequence = sequence + pos_enc
return sequence
# Test
np.random.seed(42)
patches = np.random.randn(196, 768)
seq = vit_sequence_construction(patches, D=768)
assert seq.shape == (197, 768), "Sequence shape (196 patches + 1 CLS)"
print("✓ ViT sequence construction working")
if __name__ == "__main__":
print("Lab 2: VITSequence - PASSED")### Lab 3: Patch Size Effect
import numpy as np
def analyze_patch_sizes(image_size=224):
"""Analyze effect of different patch sizes"""
patch_sizes = [8, 16, 32]
results = {}
for p in patch_sizes:
num_patches = (image_size // p) ** 2
seq_length = num_patches + 1 # +1 for CLS
flops = seq_length ** 2 # Simplified: quadratic attention
results[f"patch_{p}"] = {
"num_patches": num_patches,
"seq_length": seq_length,
"attention_flops": flops
}
return results
# Test
results = analyze_patch_sizes(image_size=224)
assert len(results) == 3, "Three patch sizes"
assert results["patch_16"]["num_patches"] == 196, "196 patches for 16x16"
assert results["patch_8"]["seq_length"] > results["patch_16"]["seq_length"], "Smaller patches → more tokens"
print("✓ Patch size analysis working")
if __name__ == "__main__":
print("Lab 3: PatchSizeAnalysis - PASSED")### Lab 4: Multi-Scale ViT
import numpy as np
def multi_scale_vit_embeddings(image, patch_sizes=[16, 32]):
"""Multi-scale patch embeddings"""
embeddings_per_scale = {}
for p in patch_sizes:
H, W, C = image.shape
patches = []
for i in range(0, H, p):
for j in range(0, W, p):
patch = image[i:i+p, j:j+p, :]
if patch.shape[:2] == (p, p):
patches.append(patch.flatten())
if patches:
patches = np.array(patches)
# Project to 768-D
W_proj = np.random.randn(patches.shape[1], 768)
embeddings = np.dot(patches, W_proj)
embeddings_per_scale[f"scale_{p}"] = embeddings
return embeddings_per_scale
# Test
np.random.seed(42)
image = np.random.randn(224, 224, 3)
embeddings = multi_scale_vit_embeddings(image, patch_sizes=[16, 32])
assert len(embeddings) == 2, "Two scales"
assert embeddings["scale_16"].shape[1] == 768, "Embedding dimension"
assert embeddings["scale_16"].shape[0] > embeddings["scale_32"].shape[0], "Smaller patches → more tokens"
print("✓ Multi-scale ViT working")
if __name__ == "__main__":
print("Lab 4: MultiScaleViT - PASSED")