neighborhood attention
**Neighborhood Attention** is the **locally dynamic attention pattern that slides a window over the feature map while keeping each center token attention-aware of its immediate neighbors** — unlike fixed convolution kernels, the attention weights change with content, so edges, textures, and small objects get adaptive emphasis without computing full global maps.
**What Is Neighborhood Attention?**
- **Definition**: A structured attention that restricts each query to a fixed-size K×K neighborhood around itself, optionally grouping the neighborhood into horizontal and vertical stripes for efficiency.
- **Key Feature 1**: Windows overlap so that every pixel participates as both query and key, ensuring continuity.
- **Key Feature 2**: Dynamic weights are recalculated per token, making the operation content-sensitive rather than purely geometric.
- **Key Feature 3**: Block-sparse implementation exploits the regular grid to batch gather operations effectively on GPUs.
- **Key Feature 4**: Windows can grow with depth or be dilated to widen the effective receptive field.
**Why Neighborhood Attention Matters**
- **Local Detail**: Maintains the precise local structure necessary for segmentation, detection, and medical imaging.
- **Cost Control**: Complexity is O(HWk^2) with small k (e.g., 7), so it scales linearly with the number of patches.
- **Receptive Field Expansion**: Dilation or stacking multiple layers gradually expands the contextual footprint without global cost.
- **Smooth Transitions**: Overlapping windows prevent blocking artifacts because tokens appear in multiple local neighborhoods.
- **Plug-In Flexibility**: Can replace Swin or other windowed attention modules without rewiring offset computations.
**Neighborhood Configurations**
**Regular Window**:
- Use K=3 or 5, with padding to keep spatial shapes constant.
- Each token attends to its immediate neighbors in a square layout.
**Dilated Neighborhood**:
- Skip tokens using dilation factor d, allowing the kernel to cover a broader area while still remaining local.
- Good for high-resolution tasks where receptive field must grow smoothly.
**Grouped Neighborhoods**:
- Partition channels or heads into groups focusing on different neighborhood ranges (e.g., near vs far neighbors).
**How It Works / Technical Details**
**Step 1**: Extract the K×K patch around each query via unfolding or strided gather, producing per-query keys and values.
**Step 2**: Compute attention scores using scaled dot product, apply softmax within the patch, and aggregate the values. Optionally add relative positional biases to encode spatial shifts.
**Comparison / Alternatives**
| Aspect | Neighborhood | Swin (Window) | Global |
|--------|--------------|---------------|--------|
| Context | Local adaptive | Local static geometry | Global |
| Learnable | Yes | Only via biases | Yes
| Blocking | Negative but mitigated by overlap | Possible if shifts missing | None
| Complexity | O(Nk^2) | O(Nw^2) | O(N^2)
**Tools & Platforms**
- **MMCViT / timm**: Provide NeighborhoodAttention modules with dilation controls.
- **MMSegmentation**: Uses neighborhood attention for high-resolution segmentation heads.
- **Detectron2**: Supports custom attention heads for both heads and FPN features.
- **TVM / Triton**: Can compile neighborhood attention kernels for efficient inference on GPUs.
Neighborhood attention is **the adaptive local focus that keeps transformers precise on small structures without blowing up compute** — it gives every token a neighborhood-aware view while keeping the cost linear with image size.