axial attention in vit
**Axial Attention** is the **factorized attention strategy that alternates row-wise and column-wise self-attention to cover entire images without quadratic compute** — by sweeping first along the height axis and then along the width axis, the layer retains full-field context while shrinking complexity to O(HW(H+W)), which lets Vision Transformers scale to megapixel inputs for satellite, microscopy, and clinical imagery without blowing up memory.
**What Is Axial Attention?**
- **Definition**: A transformer block that splits multi-head attention into two sequential passes, one attending to each row and the other attending to each column, with interleaved projections and residual merges.
- **Key Feature 1**: Row pass aggregates information within each horizontal stripe of patches while keeping positional bias along the other axis constant.
- **Key Feature 2**: Column pass then propagates those summaries vertically so every pixel eventually receives contributions from all directions.
- **Key Feature 3**: Multi-head projections in each pass reuse the same heads so parameter count stays similar to standard attention.
- **Key Feature 4**: Relative or axial positional encodings keep track of sequence order along the active axis without full 2D tables.
**Why Axial Attention Matters**
- **Resolution Scalability**: Complexity reduces from quadratic in HW to linear in the sum (H+W), enabling 1,000+ patch grids.
- **Hardware Friendliness**: Each pass performs dense matrix multiplications of shape (N, C) rather than (N, N), keeping GPU memory stable.
- **Global Receptive Field**: Alternating passes allow even distant patches to exchange information in two hops, preserving global context.
- **Gradient Stability**: Two smaller attention operations avoid the extreme softmax behavior of a single huge matrix, improving training stability.
- **Fine-Grain Control**: Designers can mix axis order or skip one axis occasionally for dynamic sparsity without rewiring the entire backbone.
**Axis Configurations**
**Row-then-Column**:
- **Row Stage**: Attends to H long sequences of length W, capturing textures and horizontal edges.
- **Column Stage**: Attends to W sequences of length H, aggregating vertical context.
- **Fusion**: Residual addition merges both stages before the feedforward sublayer.
**Column-then-Row**:
- **Order Swap**: Useful when vertical semantics dominate (e.g., document pages).
- **Symmetry**: Maintains the same compute budget with axes swapped.
**Hybrid**:
- **Local Axial Blocks**: Combine with window attention to focus networks on both near neighbors and distant patches by alternating axial/global passes every few layers.
**How It Works**
**Step 1**: Project tokens to queries, keys, and values and reshape them into (axis_length, channel), then run the first attention pass along rows, normalizing by sqrt(dk) and applying softmax with per-row masks.
**Step 2**: Feed row outputs into the second pass that attends along columns, optionally including learned relative offsets, before adding the standard feed-forward module and layer norm.
**Comparison / Alternatives**
| Aspect | Axial | Global (Full) | Window + Shift |
|--------|-------|---------------|----------------|
| Complexity | O(HW(H+W)) | O((HW)^2) | O(HWw^2) with window size w |
| Receptive Field | Two-hop global | Direct global | Patch-clustered, requires shifts |
| Memory Pressure | Linear | Quadratic | Moderate |
| Best Use Case | Gigapixel scenes | Moderate-resolution tasks | Efficiency + locality |
**Tools & Platforms**
- **PyTorch / timm**: AxialTransformer and ViT variants expose axial_config dictionaries for quick swapping.
- **DeiT / Timm scripts**: Support axial blocks as drop-in replacements for standard attention.
- **DeepSpeed / Fairscale**: Mesh-Tensor-Parallel training runs axial blocks with large batch support.
- **Model Zoo**: Axial-DeepLab and Axial-ResNet use the same axis-splitting principle outside of pure transformers.
Axial attention is **the existential tool for scaling transformers to dense, high-resolution imaging tasks** — it keeps every patch in play without ever materializing an enormous attention matrix, so practical deployments can see the whole field without compromising training budgets.