page-attention

**Page-attention** is the **paged attention mechanism that stores KV cache in fixed-size memory blocks to reduce fragmentation and enable efficient dynamic batching** - it is a key innovation in high-throughput LLM serving systems. **What Is Page-attention?** - **Definition**: Attention runtime that manages KV tensors using virtual-memory-like paging abstractions. - **Memory Layout**: Requests map to pages, allowing non-contiguous physical allocation and fast reuse. - **Serving Benefit**: Reduces allocator overhead and improves utilization under variable request lengths. - **System Context**: Commonly associated with modern engines such as vLLM. **Why Page-attention Matters** - **Fragmentation Reduction**: Fixed pages avoid severe memory waste from variable-length sequences. - **Batching Efficiency**: Paged layouts support dynamic request interleaving and continuous batching. - **Latency Stability**: Allocator predictability lowers p95 spikes under mixed traffic. - **Capacity Gains**: Higher effective memory utilization increases concurrent session count. - **Operational Simplicity**: Page-level accounting improves debugging and resource governance. **How It Is Used in Practice** - **Page Size Tuning**: Choose block size based on sequence profile and hardware characteristics. - **Runtime Instrumentation**: Track page occupancy, churn, and compaction pressure. - **Scheduler Integration**: Coordinate paging with batching and priority policies for stable throughput. Page-attention is **a foundational memory-management technique for modern inference engines** - paged attention enables scalable decode throughput with better memory utilization.