attention sink
**Attention Sinks and StreamingLLM** are the **architectural phenomenon and inference technique where the first few tokens in a sequence consistently receive disproportionately high attention regardless of content** — a pattern observed across virtually all Transformer models where initial tokens act as "attention sinks" that absorb excess attention mass, and the StreamingLLM method exploits this discovery to enable theoretically infinite context streaming by maintaining only the attention sink tokens plus a sliding window of recent tokens, providing constant-memory inference without quality degradation for indefinitely long conversations.
**The Attention Sink Phenomenon**
```
Observation: In virtually ALL transformers:
Token 0 (BOS or first word) receives 20-50% of attention mass
Token 1-3: Also receive elevated attention (5-15% each)
Remaining tokens: Share the rest proportionally to relevance
Why?
Softmax must sum to 1.0 across all tokens
When no token is particularly relevant, attention mass must go SOMEWHERE
First tokens become "default dump" for excess attention
This happens REGARDLESS of the content of those tokens
```
**Why Attention Sinks Exist**
| Hypothesis | Explanation | Evidence |
|-----------|-----------|---------|
| Positional bias | Position 0 always encountered in training | Sinks appear even with randomized positions |
| Softmax constraint | Attention must sum to 1, needs a "trash" bin | Adding a learnable sink token reduces effect |
| Token frequency | BOS/common words seen most in training | Replacing BOS with rare token still creates sink |
| Information vacuum | Early tokens have minimal conditional context | Consistent across architectures |
**StreamingLLM**
```
Problem: Standard sliding window attention fails catastrophically
Window = tokens [101-200] (dropped tokens 0-100)
Model expects attention sinks at positions 0-3 → they're gone →
Attention distribution collapses → quality tanks
StreamingLLM solution:
Keep: [Token 0, 1, 2, 3] (attention sinks) + [last N tokens] (recent context)
Drop: Everything in between
Example with window=4 sinks + 1000 recent:
Context at step 5000: [0,1,2,3] + [4001,4002,...,5000]
Context at step 50000: [0,1,2,3] + [49001,49002,...,50000]
Memory: Always constant (1004 tokens)
Quality: Comparable to full attention for recent-context tasks
```
**Perplexity Comparison**
| Method | Context | Memory | Perplexity |
|--------|---------|--------|------------|
| Full attention (ideal) | All tokens | O(N) | Baseline |
| Sliding window (no sinks) | Last 2048 | O(2048) | Explodes after window fill |
| StreamingLLM (4 sinks + 2048) | 4 + last 2048 | O(2052) | Stable, ~baseline |
| Sliding window (no sinks) failure | Last 2048 | O(2048) | >1000 PPL (broken) |
**Dedicated Attention Sink Token**
```python
# Training with a learnable sink token (prevents reliance on BOS)
class AttentionSinkModel(nn.Module):
def __init__(self, base_model):
super().__init__()
self.model = base_model
# Learnable sink token prepended to every sequence
self.sink_token = nn.Parameter(torch.randn(1, 1, d_model))
def forward(self, x):
# Prepend sink token
sink = self.sink_token.expand(x.size(0), -1, -1)
x = torch.cat([sink, x], dim=1)
return self.model(x)[:, 1:] # remove sink from output
```
**Implications for Model Design**
- Models with explicit sink tokens: Better streaming performance.
- KV cache management: Always keep sink tokens, never evict them.
- PagedAttention: Pin sink token pages in memory.
- Positional encoding: Sink tokens should have fixed (not rotated) positions.
**Applications of StreamingLLM**
| Application | Benefit |
|------------|--------|
| Multi-hour conversations | Constant memory, no OOM |
| Real-time transcription | Process infinite audio stream |
| Log analysis | Stream through gigabytes of logs |
| Code assistance | Long coding sessions without context limits |
| Monitoring agents | Run indefinitely without memory growth |
**Limitations**
- No recall of dropped tokens: Information between sinks and window is lost forever.
- Not a replacement for long context: Tasks requiring full document understanding still need full attention.
- Trade-off: Streaming capability vs. information retention.
Attention sinks and StreamingLLM are **the key insight enabling infinite-length Transformer inference** — by discovering that Transformers rely on initial tokens as attention reservoirs and preserving them alongside a sliding window, StreamingLLM provides constant-memory inference that runs indefinitely without quality collapse, solving a practical deployment problem for any application where conversations or data streams can grow without bound.