attention distance analysis

**Attention Distance** is a **quantitative, diagnostic metric that measures the average physical spatial distance (in pixels or patch positions) between the Query patch and the patches it attends to most strongly — revealing how far across the image each attention head "reaches" at every layer of a Vision Transformer and exposing the fundamental difference in receptive field behavior between ViTs and Convolutional Neural Networks.** **The Measurement Protocol** - **The Calculation**: For each attention head in each layer, the algorithm computes the weighted average distance between the Query token's spatial position and all Key token positions, weighted by the Softmax attention probabilities. If a head assigns high attention to distant patches, the attention distance is large (global). If it focuses on immediate neighbors, the distance is small (local). **The Empirical Findings** - **Lower Layers (Layers 1-4)**: Attention heads exhibit a striking mixture of behaviors. Some heads have very short attention distances, essentially mimicking the local spatial filtering behavior of early convolutional layers (detecting edges and textures in the immediate neighborhood). Other heads in the same layer simultaneously exhibit very long attention distances, attending to semantically related patches across the entire image. - **Higher Layers (Layers 8-12)**: Nearly all attention heads converge to predominantly global (long-distance) attention, aggregating high-level semantic information from across the full image extent. **The Critical Comparison with CNNs** - **CNNs (Strictly Local)**: In a ResNet, the receptive field at the very first layer is exactly $3 imes 3$ pixels. It is physically impossible for the first convolutional layer to see anything beyond its immediate 9-pixel neighborhood. Global context is only achieved after stacking dozens of layers. - **ViTs (Flexible from Layer 1)**: The Self-Attention mechanism grants every head the mathematical freedom to attend globally from the very first layer. The remarkable finding is that despite having this freedom, many early-layer heads voluntarily learn short-distance, local attention patterns, effectively rediscovering convolutional filtering from scratch (the "ConvMimic" phenomenon). **Why Attention Distance Matters** This diagnostic reveals whether a ViT is actually utilizing its global attention capability or is wasting computational resources on purely local operations that a simple convolution could perform far more efficiently. It directly motivates hybrid architectures (like LeViT or CoAtNet) that explicitly use convolutions for the first few local-dominant layers and switch to Self-Attention only for the later global-dominant layers. **Attention Distance** is **the reach map of intelligence** — measuring exactly how far each attention head stretches its sensory arms across the image, revealing whether the Transformer is truly leveraging its global vision or merely imitating a convolutional filter.