flashinfer

**FlashInfer** is an open-source library providing **highly optimized GPU kernels** specifically designed for LLM inference workloads. Developed with a focus on **flexibility and performance**, it addresses the key computational bottlenecks in serving large language models, particularly the **attention mechanism**. **Core Capabilities** - **FlashAttention for Inference**: Implements memory-efficient attention kernels optimized specifically for the **decode phase** of LLM inference, where the query length is 1 but the KV cache can be very long. - **Paged KV Cache Support**: Native support for **paged attention** — managing the key-value cache in non-contiguous memory blocks, similar to how operating systems manage virtual memory. - **Ragged Tensors**: Efficiently handles **variable-length sequences** within a batch without padding, maximizing GPU utilization when requests have different context lengths. - **Custom Attention Variants**: Supports **grouped-query attention (GQA)**, **multi-query attention (MQA)**, **sliding window attention**, and other modern attention patterns used by different model architectures. **Performance Advantages** - **Kernel Specialization**: Unlike general-purpose attention libraries, FlashInfer's kernels are specifically tuned for the **asymmetric** compute patterns of inference (short query, long KV cache). - **Composable API**: Provides building-block kernels that serving frameworks can combine and customize for their specific needs. **Integration** FlashInfer is used as a **backend kernel library** by several popular LLM serving frameworks, including **SGLang** and **vLLM**, where it provides the low-level attention computation. Rather than being an end-to-end serving solution, FlashInfer focuses on being the **fastest possible attention kernel** that other systems can build upon. It supports NVIDIA GPUs from **Ampere (A100) onwards** and is actively developed to support the latest hardware features and model architectures.