swin transformer - hierarchical vision
# Swin Transformer - Hierarchical Vision
## Introduction & Motivation
Swin Transformer: hierarchical transformer with shifted windows. Local attention for efficiency. Applications: vision tasks across resolutions, detection, segmentation.
Motivation: Enable transformers for downstream vision tasks efficiently.
Applications: Object detection, segmentation, classification.
---
## Core Concepts & Theory
### Shifted Windows
Local attention within windows.
### Hierarchical Structure
Multi-scale feature pyramid.
### Window Attention
Reduce attention complexity.
### Shifted Attention
Shift window pattern alternately.
---
## Mathematical Formulation
Window Attention:
$$ ext{Attention}(Q, K, V) = ext{softmax}(\frac{QK^T}{\sqrt{d}} + B) V$$
Window Complexity:
$$O(4hw(M^2 + 4M)) \approx O(hwM^2)$$
Shifted Attention:
$$ ext{pattern}_{ ext{even}}
eq ext{pattern}_{ ext{odd}}$$
---
## Advanced Theory & Extensions
### Continuous Shifting
Smooth attention transitions.
### Depthwise Attention
Separate attention per window.
### Cross-Window Connections
Connect adjacent windows.
---
## Computational Considerations
Window attention: O(w²·D²).
Shifted windows: O(4·w²·D²).
Total: ~4× less than ViT.
---
## Practical Implementation Strategies
### Window Size Selection
Typical 7×7 windows.
### Shift Magnitude
Half of window size.
### Multi-Head Attention
8-12 heads per layer.
---
## Benchmark Datasets & Evaluation
ImageNet: Classification benchmark.
COCO: Detection benchmark.
ADE20K: Segmentation benchmark.
---
## Key Challenges & Limitations
### Window Complexity
Careful shift implementation.
### Cross-Window Dependencies
Limited long-range context.
### Training Stability
Careful initialization.
---
## Hyperparameter Tuning
Window size: 7-14.
Shift amount: Window size / 2.
Number of heads: 8-12.
---
## Real-World Applications & Case Studies
Object Detection: Strong backbone for FPN.
Semantic Segmentation: Hierarchical features.
Image Classification: Transfer learning baseline.
---
## Integration with Other Methods
Swin + FPN for detection; + multi-scale features.
---
## Summary & Key Takeaways
Swin Transformer enables efficient hierarchical vision.
Principles:
1. Shifted windows: Local attention.
2. Hierarchical: Multi-scale features.
3. Efficiency: Reduced complexity.
4. Stacking: Shifted alternately.
5. Versatility: Diverse downstream tasks.
---
## Appendix: Practical Labs
### Lab 1: Window Partition
import numpy as np
def window_partition(x, window_size=7):
"""Partition feature map into windows"""
b, h, w, c = x.shape
x = x.reshape(b, h // window_size, window_size, w // window_size, window_size, c)
x = x.transpose(0, 1, 3, 2, 4, 5)
windows = x.reshape(-1, window_size, window_size, c)
return windows
np.random.seed(42)
feat = np.random.randn(1, 224, 224, 96)
windows = window_partition(feat, 7)
assert windows.shape[0] == (224//7) ** 2
print(f"✓ Window partition: {windows.shape}")### Lab 2: Window Attention
import numpy as np
def window_attention(x, window_size=7, num_heads=8):
"""Apply attention within windows"""
b, h, w, c = x.shape
# Partition into windows
windows = []
for i in range(0, h, window_size):
for j in range(0, w, window_size):
window = x[:, i:i+window_size, j:j+window_size, :]
windows.append(window)
# Apply self-attention per window
attended = []
for window in windows:
# Simplified attention
attn_window = window # Skip actual attention for brevity
attended.append(attn_window)
return attended
np.random.seed(42)
feat = np.random.randn(1, 224, 224, 96)
attended = window_attention(feat)
assert len(attended) == (224//7) ** 2
print("✓ Window attention working")### Lab 3: Shifted Window
import numpy as np
def shift_window(x, shift_size=3):
"""Shift window for alternating pattern"""
b, h, w, c = x.shape
shifted = np.roll(x, shift_size, axis=1)
shifted = np.roll(shifted, shift_size, axis=2)
return shifted
np.random.seed(42)
feat = np.random.randn(1, 224, 224, 96)
shifted = shift_window(feat, 3)
assert shifted.shape == feat.shape
assert not np.allclose(feat, shifted)
print("✓ Shifted window working")### Lab 4: Hierarchical Features
import numpy as np
def hierarchical_features(x, num_stages=4):
"""Extract hierarchical features from Swin"""
features = []
current = x
for stage in range(num_stages):
# Downsample
if stage > 0:
current = current[:, ::2, ::2, :] # 2x downsample
features.append(current)
return features
np.random.seed(42)
x = np.random.randn(1, 224, 224, 96)
feats = hierarchical_features(x)
assert len(feats) == 4
assert feats[0].shape[1] == 224
assert feats[-1].shape[1] == 224 // 8
print("✓ Hierarchical features working")---