Landmark Attention is the efficient transformer attention mechanism that reduces computational complexity by routing all token attention through a sparse set of landmark (anchor) tokens that serve as information hubs — achieving sub-quadratic attention cost while preserving global information flow — the architecture that demonstrates how strategically placed landmark tokens can serve as a compressed global context, enabling long-sequence processing without the full O(n²) cost of standard self-attention.

What Is Landmark Attention?

• Definition: A modified attention mechanism where regular tokens attend only to nearby local tokens and to a set of specially designated landmark tokens, while landmark tokens attend to all other landmarks — creating a two-level attention hierarchy with O(n × k) complexity where k << n is the number of landmarks.
• Landmark Selection: Landmarks are chosen at fixed intervals (every m-th token), at content boundaries (sentence/paragraph breaks), or through learned prominence scoring — they serve as representative summaries of their local region.
• Two-Level Attention: (1) Local tokens attend to their neighborhood + all landmarks (sparse), (2) Landmarks attend to all other landmarks (dense but small) — global information propagates through the landmark network while local processing remains efficient.
• Information Bridge: Landmarks act as bridges between distant sequence regions — a token at position 1 can influence a token at position 10,000 through their respective nearest landmarks, which are connected via landmark-to-landmark attention.

Why Landmark Attention Matters

• Sub-Quadratic Complexity: Standard attention is O(n²); Landmark attention is O(n × k + k²) where k << n — for k = √n, this becomes O(n^1.5), dramatically more efficient for long sequences.
• Global Information Preservation: Unlike local-only attention (which loses distant context), landmark-to-landmark attention maintains a global information pathway — important for tasks requiring full-document understanding.
• Minimal Quality Loss: Well-placed landmarks preserve 95%+ of full attention's information — the compression through landmarks retains the most important global signals.
• Compatible With Flash Attention: The local attention windows and landmark attention patterns can be implemented efficiently with existing optimized kernels.
• Configurable Trade-Off: Adjusting landmark density (k) provides a smooth trade-off between efficiency and information retention — more landmarks = more global information at higher cost.

Landmark Attention Architecture

Landmark Placement Strategies:

• Fixed Stride: Every m-th token is a landmark — simplest, works well for uniform-density text.
• Learned Selection: A scoring network assigns prominence scores; top-k scoring tokens become landmarks — content-aware, better for heterogeneous inputs.
• Boundary-Based: Landmarks placed at sentence boundaries, paragraph breaks, or topic transitions — aligns with natural information structure.

Attention Pattern:

• Regular token t attends to: local window [t−w, t+w] UNION all landmarks.
• Landmark l attends to: its local region UNION all other landmarks.
• This creates a sparse attention pattern with guaranteed global connectivity.

Complexity Comparison

Landmark Attention is the information-routing architecture that proves global context can be maintained through strategic compression — using a sparse network of landmark tokens as information hubs that connect distant sequence regions at sub-quadratic cost, achieving the practical efficiency of local attention with the semantic capability of global attention.