Shifted window attention is the cross-window communication mechanism in Swin Transformer that shifts the window partition grid by half a window size between consecutive transformer layers — enabling information flow across window boundaries while maintaining the computational efficiency of local window attention, effectively providing global context through alternating local computations.
What Is Shifted Window Attention?
- Definition: A technique where the spatial partitioning of attention windows is offset by (M/2, M/2) pixels between consecutive transformer layers, so that tokens at the boundary of one layer's windows are placed in the interior of the next layer's windows, enabling cross-boundary information exchange.
- Swin Transformer Core: The defining innovation of the Swin Transformer (Liu et al., 2021, "Hierarchical Vision Transformer using Shifted Windows") that solves the isolation problem of non-overlapping window attention.
- Alternating Pattern: Layer L uses regular window partition. Layer L+1 shifts the partition by (⌊M/2⌋, ⌊M/2⌋). Layer L+2 returns to regular partition. This alternation continues through all layers.
- Effective Global Receptive Field: After just a few alternating layers, information can propagate across the entire image through successive cross-window connections.
Why Shifted Window Attention Matters
- Breaks Window Isolation: Without shifting, tokens in different windows can never interact — shifted windows create bridges between previously isolated regions.
- Maintains Linear Complexity: The shifting operation itself has zero computational cost — it's just a change in how tokens are grouped, not an additional attention computation.
- Equivalent to Cross-Window Attention: Mathematically, alternating regular and shifted windows achieves similar information flow to overlapping windows or cross-window attention, but with lower implementation complexity.
- Hierarchical Global Context: Combined with patch merging (spatial downsampling), shifted windows enable global context to emerge naturally — early layers handle local features, later layers (with reduced spatial resolution) handle global relationships.
- SOTA Performance: Swin Transformer with shifted window attention achieved state-of-the-art results on ImageNet classification, COCO detection, and ADE20K segmentation upon release.
How Shifted Window Attention Works
Regular Window (Layer L):
- Feature map partitioned into non-overlapping M×M windows.
- Example with M=4 on an 8×8 map: 4 windows, each 4×4 = 16 tokens.
- Self-attention computed independently within each window.
Shifted Window (Layer L+1):
- Window grid shifted by (M/2, M/2) = (2, 2) pixels.
- New window boundaries now cross the centers of the previous windows.
- Tokens that were at the edges of regular windows are now in the middle of shifted windows.
- Cross-boundary information flows naturally through attention within the new windows.
Efficient Masking Implementation:
- Naive shifting creates irregular windows at image borders (different sizes).
- Cyclic Shift: Instead of padding, the feature map is cyclically shifted, creating full-size windows everywhere.
- Attention Mask: A mask prevents tokens from different original spatial regions from attending to each other within the same shifted window.
- This approach maintains a uniform window count and avoids padding overhead.
Information Flow Example
| Layer | Window Config | Cross-Window Info |
|-------|-------------|-------------------|
| Layer 1 | Regular windows | None — isolated |
| Layer 2 | Shifted windows | Adjacent windows connected |
| Layer 3 | Regular windows | 2-hop connections form |
| Layer 4 | Shifted windows | 3-hop connections — near global |
| Layer 5+ | Alternating | Effectively global receptive field |
Swin Transformer Architecture
| Stage | Layers | Window | Resolution | Cross-Window |
|-------|--------|--------|-----------|-------------|
| Stage 1 | 2 | 7×7 | 56×56 | 1 shifted layer |
| Stage 2 | 2 | 7×7 | 28×28 | 1 shifted layer |
| Stage 3 | 6-18 | 7×7 | 14×14 | 3-9 shifted layers |
| Stage 4 | 2 | 7×7 | 7×7 | 1 shifted layer (global) |
Performance Impact
| Model | Attention Type | ImageNet Top-1 | FLOPs |
|-------|---------------|----------------|-------|
| ViT-B/16 | Global | 77.9% | 17.6G |
| DeiT-B | Global + distill | 83.4% | 17.6G |
| Swin-B | Shifted window | 83.5% | 15.4G |
| Swin-L | Shifted window | 87.3% | 34.5G |
Shifted window attention is the elegant solution to the locality-efficiency tradeoff in Vision Transformers — by simply alternating window positions between layers, Swin Transformer achieves global information flow with purely local computation, proving that cleverness in architecture design can be more powerful than brute-force compute.