local vs global attention in vit
**Local vs global attention in ViT** is the **design tradeoff between restricted neighborhood focus and full image token interactions when building efficient transformer vision models** - local attention reduces compute and often improves detail modeling, while global attention captures long-range relationships directly.
**What Is the Local vs Global Attention Tradeoff?**
- **Local Attention**: Each token attends to nearby patches inside a window.
- **Global Attention**: Each token attends to all tokens in the image sequence.
- **Complexity Impact**: Local patterns scale near linearly, global patterns scale quadratically.
- **Model Behavior**: Local improves fine textures, global improves scene-level context.
**Why This Tradeoff Matters**
- **Scalability**: High-resolution workloads are often impossible with pure global attention.
- **Accuracy Balance**: Pure local can miss distant dependencies, pure global can waste compute.
- **Architecture Choice**: Many modern backbones alternate local and occasional global blocks.
- **Deployment Fit**: Edge deployment often favors local windows with sparse global refresh.
- **Task Specificity**: Detection and segmentation usually need stronger local detail pathways.
**Common Design Patterns**
**Windowed Local Blocks**:
- Use fixed K x K windows for efficient neighborhood modeling.
- Add shifted windows between blocks to share cross-window context.
**Periodic Global Blocks**:
- Insert full attention at intervals to propagate global semantics.
- Maintains long-range coherence with bounded cost.
**Hybrid Heads**:
- Some heads attend locally while others attend globally in same layer.
- Improves representational diversity.
**Practical Guidance**
- **High Resolution Inputs**: Start with local attention baseline, then add sparse global layers.
- **Global Context Tasks**: Keep enough global blocks for scene-level reasoning.
- **Profiling First**: Measure FLOPs and memory before deciding hybrid depth ratio.
Local vs global attention in ViT is **a central efficiency and quality lever that defines how a model spends its compute budget** - good hybrid design delivers near-global understanding without quadratic runtime penalties.