landmark attention

**Landmark attention** is the **attention strategy that introduces selected anchor tokens or summary landmarks to help models access long-range information efficiently** - it reduces full quadratic attention cost while preserving global context access paths. **What Is Landmark attention?** - **Definition**: Sparse attention design where regular tokens attend through designated landmark nodes. - **Mechanism**: Landmark tokens act as compressed hubs for long-range information routing. - **Complexity Benefit**: Cuts attention compute relative to dense all-to-all attention. - **Long-Context Role**: Supports longer sequences by improving memory and compute scalability. **Why Landmark attention Matters** - **Efficiency**: Enables longer inputs under fixed hardware budgets. - **Global Access**: Maintains pathways for distant dependency handling. - **RAG Relevance**: Useful when prompts include many retrieved chunks and long histories. - **Architectural Flexibility**: Can be combined with other sparse or hierarchical attention methods. - **Tradeoff Management**: Requires careful landmark design to avoid information bottlenecks. **How It Is Used in Practice** - **Landmark Selection**: Choose anchors by structure boundaries, salience scores, or learned policies. - **Hybrid Attention**: Blend local dense windows with landmark-mediated global connections. - **Task Benchmarks**: Evaluate long-range reasoning, factuality, and latency before deployment. Landmark attention is **an efficient long-context attention pattern for scalable transformers** - well-chosen landmarks preserve global reasoning while reducing computational burden.