Semantic Segmentation Transformer Models
# Semantic Segmentation & Transformer Models
## Introduction & Motivation
Semantic Segmentation with Transformers: pixel-level classification using attention. Vision Transformers (ViT), SETR. Applications: scene understanding, medical imaging.
Motivation: Leverage transformer efficiency for dense prediction.
Applications: Autonomous driving, medical imaging, scene parsing.
---
## Core Concepts & Theory
### Vision Transformers
Patch-based image processing.
### Self-Attention
Global context in segmentation.
### SETR
Sequence-to-sequence for segmentation.
### Cross-Attention
Query encoder-decoder architecture.
---
## Mathematical Formulation
Vision Transformer Patch Embedding:
$$z_0 = [x_p^1E; x_p^2E; ...; x_p^NE] + E_ ext{pos}$$
Self-Attention:
$$A = ext{softmax}(\frac{QK^T}{\sqrt{d}})V$$
Segmentation Head:
$$p(y_i|x) = ext{softmax}(W h_i + b)$$
---
## Advanced Theory & Extensions
### Swin Transformer
Hierarchical vision transformer.
### SegFormer
Efficient transformer for segmentation.
### SegViT
Vision transformer variants.
---
## Computational Considerations
Patch embedding: O(N·patch²).
Self-attention: O(T²·d).
Classification head: O(H·W·C).
---
## Practical Implementation Strategies
### Hierarchical Architectures
Multi-scale feature extraction.
### Efficient Attention
Window-based, shifted attention.
### Pre-training
ImageNet pre-training transfer.
---
## Benchmark Datasets & Evaluation
ADE20K: 150 classes indoor segmentation.
Cityscapes: 19 classes urban scenes.
Pascal VOC: 21 classes segmentation.
---
## Key Challenges & Limitations
### Computational Cost
High memory and computation.
### Small-Scale Patterns
Transformer receptive fields.
### Data Requirements
Large datasets for good performance.
---
## Hyperparameter Tuning
Patch size: 16-32 pixels.
Attention heads: 8-16.
Learning rate: 1e-5 to 1e-4.
---
## Real-World Applications & Case Studies
Autonomous Driving: Road scene segmentation.
Medical Imaging: Organ segmentation.
Aerial Imagery: Land-use classification.
---
## Integration with Other Methods
Semantic segmentation + instance detection for panoptic segmentation; + post-processing for boundary refinement.
---
## Summary & Key Takeaways
Semantic Segmentation via transformers enables efficient global context modeling.
Principles:
1. Patch embedding: Tokenization.
2. Self-attention: Global dependencies.
3. Hierarchical features: Multi-scale.
4. Decoder architecture: Dense prediction.
5. Efficient variants: Computational efficiency.
---
---
## Appendix: Practical Labs
### Lab 1: Patch Embedding
import numpy as np
def patch_embedding(image, patch_size=16):
"""Extract patch embeddings"""
h, w, c = image.shape
patches = []
for i in range(0, h, patch_size):
for j in range(0, w, patch_size):
patch = image[i:i+patch_size, j:j+patch_size]
patch_flat = patch.flatten()
patches.append(patch_flat)
return np.array(patches)
# Test
np.random.seed(42)
image = np.random.rand(224, 224, 3)
patches = patch_embedding(image, patch_size=16)
n_patches = (224 // 16) ** 2
assert len(patches) == n_patches, "Correct patch count"
print("✓ Patch embedding working")
if __name__ == "__main__":
print("Lab 1: PatchEmbedding - PASSED")### Lab 2: Positional Encoding
import numpy as np
def add_positional_encoding(patches, d_model=768):
"""Add positional encoding to patches"""
n_patches, patch_dim = patches.shape
pos_encoding = np.zeros((n_patches, d_model))
for i in range(n_patches):
for j in range(0, d_model, 2):
pos_encoding[i, j] = np.sin(i / (10000 ** (j / d_model)))
if j + 1 < d_model:
pos_encoding[i, j+1] = np.cos(i / (10000 ** ((j+1) / d_model)))
return pos_encoding
# Test
np.random.seed(42)
patches = np.random.rand(196, 768)
pos_enc = add_positional_encoding(patches)
assert pos_enc.shape == patches.shape, "Correct encoding shape"
print("✓ Positional encoding working")
if __name__ == "__main__":
print("Lab 2: PositionalEncoding - PASSED")### Lab 3: Self-Attention
import numpy as np
def self_attention(query, key, value, d_k=None):
"""Compute self-attention"""
if d_k is None:
d_k = query.shape[-1]
# Scaled dot-product
scores = query @ key.T / np.sqrt(d_k)
# Softmax
attn_weights = np.exp(scores) / np.sum(np.exp(scores), axis=1, keepdims=True)
# Context
context = attn_weights @ value
return context, attn_weights
# Test
np.random.seed(42)
q = np.random.rand(196, 64)
k = np.random.rand(196, 64)
v = np.random.rand(196, 64)
context, weights = self_attention(q, k, v)
assert context.shape == v.shape, "Correct context shape"
print("✓ Self-attention working")
if __name__ == "__main__":
print("Lab 3: SelfAttention - PASSED")### Lab 4: Segmentation Head
import numpy as np
def segmentation_head(features, num_classes):
"""Simple segmentation head"""
# Reshape features to spatial dimensions
b, h, w, c = 1, 56, 56, 768
# Linear projection to classes
seg_logits = np.random.randn(h, w, num_classes)
return seg_logits
# Test
features = np.random.rand(1, 56, 56, 768)
logits = segmentation_head(features, num_classes=19)
assert logits.shape[2] == 19, "Correct class dimension"
print("✓ Segmentation head working")
if __name__ == "__main__":
print("Lab 4: SegmentationHead - PASSED")