context length extension
**Context Length Extension** is the **set of techniques for enabling LLMs trained on short sequences to process much longer sequences at inference time** — expanding usable context from 4K to 128K, 1M, or more tokens.
**Why Context Length Matters**
- 4K tokens ≈ 3,000 words ≈ 6 pages.
- 128K tokens ≈ 100,000 words ≈ entire novel.
- Long context enables: full codebase reasoning, book summarization, long document QA, multi-turn dialogue.
**The Length Generalization Problem**
- Models trained on 4K sequences struggle with 8K at inference — position IDs out-of-distribution.
- Attention scores become noisy at long ranges not seen during training.
- RoPE frequencies need adjustment for longer contexts.
**Extension Techniques**
**RoPE Scaling**:
- **Linear Interpolation**: Scale position indices by context_extension / train_length. Simple, loses some accuracy.
- **NTK-Aware Scaling**: Distributes interpolation across frequency dimensions — better quality.
- **YaRN (Yet Another RoPE extensioN)**: Dynamic NTK + attention temperature scaling. Used in LLaMA 3 (128K).
- **LongRoPE**: Non-uniform RoPE rescaling per dimension — extends to 2M tokens.
**Architecture Changes**:
- **Grouped-Query Attention (GQA)**: Fewer KV heads — reduces KV cache size linearly.
- **Sliding Window Attention (Mistral)**: Each token attends to only W nearby tokens — O(NW) instead of O(N²).
**Efficient Attention for Long Contexts**:
- FlashAttention-2/3: Enables 100K+ context without OOM.
- Ring Attention: Distribute long sequences across multiple GPUs.
**KV Cache Compression**:
- **SnapKV**: Evict less-attended KV cache entries.
- **StreamingLLM**: Attend to initial tokens + recent window.
- **H2O**: Heavy-Hitter Oracle — keep most-attended keys.
Context length extension is **a critical frontier in LLM capability** — closing the gap between model context and real-world document lengths unlocks entirely new application categories.