swin transformer
**Swin Transformer** is the **hierarchical vision transformer that makes self-attention practical for high-resolution images through shifted window attention — computing attention within fixed-size local windows and enabling cross-window communication through alternating window partitions across layers** — achieving linear computational complexity with respect to image size (vs. quadratic for standard ViT), becoming the dominant backbone for dense prediction tasks (object detection, semantic segmentation) and overtaking CNNs on every major computer vision benchmark.
**What Is Swin Transformer?**
- **Hierarchical Architecture**: Like CNNs, Swin produces multi-scale feature maps by progressively merging patches — 4×, 8×, 16×, 32× downsampling stages.
- **Window Attention**: Self-attention is computed only within non-overlapping $M imes M$ windows (typically $M = 7$), reducing complexity from $O(n^2)$ to $O(n cdot M^2)$.
- **Shifted Windows**: Alternate layers shift the window partition by $(lfloor M/2
floor, lfloor M/2
floor)$ pixels — enabling information flow between adjacent windows without overlap.
- **Key Paper**: Liu et al. (2021), "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" — ICCV 2021 Best Paper.
**Why Swin Transformer Matters**
- **Linear Complexity**: Standard ViT has $O(n^2)$ attention cost for $n$ patches — prohibitive for high-resolution images (1024×1024 = 65K patches). Swin's windowed attention is $O(n)$.
- **Dense Prediction Compatibility**: The hierarchical multi-scale design produces feature pyramids that plug directly into existing detection (FPN, Faster R-CNN) and segmentation (UPerNet) frameworks.
- **Universal Backbone**: Replaced CNNs as the default backbone for nearly all vision tasks — classification, detection, segmentation, video understanding.
- **Hardware Efficiency**: Fixed window sizes enable efficient batched matrix multiplication — well-suited to GPU architecture.
- **Transfer Learning**: Pre-trained Swin features transfer exceptionally well to downstream tasks with minimal fine-tuning.
**Architecture Details**
| Stage | Resolution | Channels | Windows | Function |
|-------|-----------|----------|---------|----------|
| **Patch Embed** | H/4 × W/4 | C | - | Split image into 4×4 patches, project to C dimensions |
| **Stage 1** | H/4 × W/4 | C | 7×7 | Swin Transformer blocks with shifted window attention |
| **Stage 2** | H/8 × W/8 | 2C | 7×7 | Patch merging (2× downsample) + Swin blocks |
| **Stage 3** | H/16 × W/16 | 4C | 7×7 | Patch merging + Swin blocks |
| **Stage 4** | H/32 × W/32 | 8C | 7×7 | Patch merging + Swin blocks |
**Shifted Window Mechanism**
- **Regular Window (Layer $l$)**: Partition feature map into non-overlapping $7 imes 7$ windows. Compute self-attention within each window independently.
- **Shifted Window (Layer $l+1$)**: Shift the window partition by $(3, 3)$ pixels. Tokens that were in different windows now share a window — enabling cross-window information exchange.
- **Efficient Implementation**: Use cyclic shifting + attention masking to maintain the same number of windows (avoids padding overhead).
**Swin Variants and Successors**
- **Swin-T/S/B/L**: Tiny (29M), Small (50M), Base (88M), Large (197M) — scaling from mobile to datacenter.
- **Swin V2**: Extended to 3 billion parameters and 1536×1536 resolution with log-spaced continuous position bias and residual-post-normalization.
- **Video Swin**: Extends windows to 3D (spatial + temporal) for video understanding — state-of-the-art on video classification benchmarks.
- **CSWin**: Cross-shaped window attention for better long-range modeling within the shifted window paradigm.
Swin Transformer is **the architecture that dethroned CNNs as the default computer vision backbone** — proving that the right attention windowing strategy makes transformers not just competitive but superior to convolutional networks for every vision task, from image classification to pixel-level dense prediction.