Neighborhood Attention is the locally dynamic attention pattern that slides a window over the feature map while keeping each center token attention-aware of its immediate neighbors — unlike fixed convolution kernels, the attention weights change with content, so edges, textures, and small objects get adaptive emphasis without computing full global maps.

What Is Neighborhood Attention?

• Definition: A structured attention that restricts each query to a fixed-size K×K neighborhood around itself, optionally grouping the neighborhood into horizontal and vertical stripes for efficiency.
• Key Feature 1: Windows overlap so that every pixel participates as both query and key, ensuring continuity.
• Key Feature 2: Dynamic weights are recalculated per token, making the operation content-sensitive rather than purely geometric.
• Key Feature 3: Block-sparse implementation exploits the regular grid to batch gather operations effectively on GPUs.
• Key Feature 4: Windows can grow with depth or be dilated to widen the effective receptive field.

Why Neighborhood Attention Matters

• Local Detail: Maintains the precise local structure necessary for segmentation, detection, and medical imaging.
• Cost Control: Complexity is O(HWk^2) with small k (e.g., 7), so it scales linearly with the number of patches.
• Receptive Field Expansion: Dilation or stacking multiple layers gradually expands the contextual footprint without global cost.
• Smooth Transitions: Overlapping windows prevent blocking artifacts because tokens appear in multiple local neighborhoods.
• Plug-In Flexibility: Can replace Swin or other windowed attention modules without rewiring offset computations.

Neighborhood Configurations

Regular Window:

• Use K=3 or 5, with padding to keep spatial shapes constant.
• Each token attends to its immediate neighbors in a square layout.

Dilated Neighborhood:

• Skip tokens using dilation factor d, allowing the kernel to cover a broader area while still remaining local.
• Good for high-resolution tasks where receptive field must grow smoothly.

Grouped Neighborhoods:

• Partition channels or heads into groups focusing on different neighborhood ranges (e.g., near vs far neighbors).

How It Works / Technical Details

Step 1: Extract the K×K patch around each query via unfolding or strided gather, producing per-query keys and values.

Step 2: Compute attention scores using scaled dot product, apply softmax within the patch, and aggregate the values. Optionally add relative positional biases to encode spatial shifts.

Comparison / Alternatives

Tools & Platforms

• MMCViT / timm: Provide NeighborhoodAttention modules with dilation controls.
• MMSegmentation: Uses neighborhood attention for high-resolution segmentation heads.
• Detectron2: Supports custom attention heads for both heads and FPN features.
• TVM / Triton: Can compile neighborhood attention kernels for efficient inference on GPUs.

Neighborhood attention is the adaptive local focus that keeps transformers precise on small structures without blowing up compute — it gives every token a neighborhood-aware view while keeping the cost linear with image size.