Dilated Attention is a sparse attention pattern where each token attends to positions at regular intervals (dilation rate d) rather than consecutive positions — similar to dilated convolutions in computer vision, enabling an exponentially growing receptive field across layers when using geometrically increasing dilation rates (d=1, 2, 4, 8...), so that a token can attend to distant positions without the O(n²) cost of full attention.
What Is Dilated Attention?
- Definition: An attention pattern where token at position i attends to positions {i, i±d, i±2d, ..., i±kd} where d is the dilation rate and k determines the number of attended positions per direction. With dilation rate d=4, a token attends to every 4th position within its receptive field.
- The Inspiration: Borrowed directly from dilated (atrous) convolutions in computer vision — where WaveNet and DeepLab used geometrically increasing dilation rates to achieve large receptive fields without proportionally increasing parameters or computation.
- The Insight: By using different dilation rates at different layers (or different heads), the model builds a multi-scale view — small dilation captures local patterns, large dilation captures global patterns, and stacking them creates an exponentially large receptive field.
How Dilation Works
| Position i=20, Window=8 | Consecutive (d=1) | Dilated (d=2) | Dilated (d=4) |
|---|---|---|---|
| Attends to positions | 13-20 | 6,8,10,12,14,16,18,20 | 0,4,8,12,16,20 (within range) |
| Span covered | 8 tokens | 16 tokens | 32 tokens |
| Tokens attended | 8 | 8 | 8 (same compute) |
| Receptive field | 8 | 16 | 32 |
Same compute cost, but 2× and 4× larger receptive fields.
Multi-Scale Dilation Across Layers
| Layer | Dilation Rate | Receptive Field (w=8) | What It Captures |
|---|---|---|---|
| Layer 1 | d=1 | 8 tokens | Local syntax, adjacent words |
| Layer 2 | d=2 | 16 tokens | Phrase-level patterns |
| Layer 3 | d=4 | 32 tokens | Sentence-level context |
| Layer 4 | d=8 | 64 tokens | Paragraph-level context |
| Layer 5 | d=16 | 128 tokens | Section-level patterns |
| Layer 6 | d=32 | 256 tokens | Document-level themes |
Combined receptive field after 6 layers: covers 256 tokens while each layer attends to only 8 positions — O(n × w) total.
Dilated Attention in Multi-Head Settings
| Head | Dilation Rate | Coverage | Role |
|---|---|---|---|
| Heads 1-2 | d=1 | Dense local | Fine-grained syntax |
| Heads 3-4 | d=2 | Sparse medium range | Phrase structure |
| Heads 5-6 | d=4 | Sparse long range | Discourse relations |
| Heads 7-8 | d=8 | Very sparse, very long range | Document structure |
Different heads with different dilation rates within the same layer provide simultaneous multi-scale attention.
Models Using Dilated Attention
| Model | Implementation | How Used |
|---|---|---|
| Longformer | Dilated sliding windows in upper layers | Combined with local + global attention |
| LongNet | Dilated attention with exponential dilation | Achieved 1B token context (theoretical) |
| BigBird | Random attention (similar sparse effect) | Alternative to explicit dilation |
| Sparse Transformer | Strided attention (related pattern) | Fixed stride patterns |
Dilated Attention is a powerful technique for building multi-scale receptive fields in efficient transformers — enabling each token to attend to distant positions at regular intervals while maintaining the same compute budget as local attention, with geometrically increasing dilation rates across layers or heads creating exponentially large effective receptive fields that capture patterns from word-level to document-level without quadratic computational cost.
Explore 500+ Semiconductor & AI Topics
From EUV lithography to CUDA optimization — search the full knowledge base or chat with our AI assistant.