Local window attention is the computational efficiency strategy that restricts self-attention computation to small fixed-size local windows rather than the full image — reducing the quadratic complexity of standard global self-attention from O(N²) to O(N) linear complexity with respect to image size, making transformer processing of high-resolution images computationally feasible.
What Is Local Window Attention?
- Definition: A modified self-attention mechanism where each token only attends to other tokens within the same fixed-size spatial window (typically 7×7 or 8×8 tokens), rather than attending to every token in the entire image.
- Swin Transformer: Introduced as the core attention mechanism in the Swin Transformer (Liu et al., 2021), replacing global self-attention with window-based attention partitioned into non-overlapping local regions.
- Complexity Reduction: For an image with N patches, global attention costs O(N²) — for a 56×56 feature map (3136 tokens), that's ~9.8 million attention computations. Window attention with 7×7 windows costs O(49 × N/49 × 49) = O(49N), which is linear in N.
- Locality Principle: In natural images, nearby pixels are more correlated than distant pixels — local attention captures the most informative relationships while discarding less useful long-range computations.
Why Local Window Attention Matters
- High-Resolution Processing: Global self-attention is impractical for high-resolution images — a 1024×1024 image with 4×4 patches produces 65,536 tokens, making O(N²) attention (~4.3 billion operations) infeasible. Window attention reduces this to manageable levels.
- Linear Scaling: Compute cost scales linearly with image resolution instead of quadratically, enabling ViTs to process images at any resolution without a compute explosion.
- Dense Prediction Tasks: Object detection and segmentation require high-resolution feature maps — window attention makes transformer backbones practical for these tasks.
- Memory Efficiency: Memory usage also scales linearly instead of quadratically, enabling larger batch sizes and higher resolution training on the same hardware.
- Competitive Performance: Despite limiting attention scope, window-based transformers achieve state-of-the-art performance by combining local attention with cross-window information exchange mechanisms.
How Local Window Attention Works
Step 1 — Window Partition:
- Divide the H×W feature map into non-overlapping windows of size M×M (typically M=7).
- For a 56×56 feature map with M=7: 8×8 = 64 windows, each containing 49 tokens.
Step 2 — Independent Attention:
- Compute standard multi-head self-attention independently within each window.
- Each token attends to all M² tokens in its window.
- Cost per window: O(M⁴) in FLOPs.
Step 3 — Output Assembly:
- Reassemble the independently processed windows back into the full feature map.
- No information crosses window boundaries in this step.
Complexity Comparison
| Attention Type | Complexity | 56×56 Feature Map | 112×112 Feature Map |
|---------------|-----------|-------------------|---------------------|
| Global | O(N²) | 9.8M ops | 157M ops |
| Window (M=7) | O(M² × N) | 154K ops | 614K ops |
| Speedup | — | 64× | 256× |
Limitations and Solutions
- No Cross-Window Communication: Tokens in different windows cannot interact — solved by shifted window attention (alternating window positions between layers).
- Fixed Receptive Field: Each layer only sees M×M tokens — stacking multiple layers with shifted windows gradually expands the effective receptive field.
- Window Boundary Artifacts: Objects split across window boundaries may not be properly modeled — shifted windows and overlapping windows mitigate this.
- Global Context Missing: Some tasks require global context that pure local attention cannot provide — hybrid architectures add occasional global attention layers (e.g., every 4th layer).
Local Window Attention Variants
- Swin Transformer: Non-overlapping windows with shifted window attention for cross-window communication.
- Neighborhood Attention (NAT): Each token attends to its K nearest spatial neighbors, providing a sliding window effect.
- Dilated Window Attention: Windows with gaps (dilation) to increase receptive field without increasing window size.
- Axial Attention: Factorize 2D attention into separate row and column attention, providing global attention along each axis with linear cost.
Local window attention is the key efficiency breakthrough that made Vision Transformers practical for real-world vision tasks — by recognizing that most visual information is local, window attention achieves near-global understanding at a fraction of the computational cost.