ring attention

Ring attention distributes attention computation across multiple devices arranged in a ring topology, enabling training and inference with extremely long context lengths by overlapping communication with computation. Concept: divide the input sequence into chunks, assign each chunk to a GPU. Each GPU computes attention for its local query chunk against key/value blocks. Key/value blocks are passed around the ring so each GPU eventually attends to the full sequence. Algorithm: (1) Each GPU holds query chunk Q_i and initially its own KV chunk (K_i, V_i); (2) Compute local attention: attention(Q_i, K_i, V_i); (3) Send KV chunk to next GPU in ring, receive from previous; (4) Compute attention with received KV chunk, accumulate with online softmax; (5) Repeat N-1 times until all KV chunks have been seen; (6) Final result: each GPU has full attention output for its query chunk. Communication overlap: while computing attention on current KV block, simultaneously transfer next KV block—if compute time ≥ transfer time, communication is fully hidden. Memory efficiency: each GPU only stores its local sequence chunk (length/N) plus one KV block being transferred—O(L/N) per GPU instead of O(L). This enables sequences N× longer than single-GPU capacity. Online softmax: critical for correctness—attention outputs from different KV blocks must be correctly combined using the log-sum-exp trick to maintain numerical stability without materializing the full attention matrix. Variants: (1) Striped attention—reorder tokens so each chunk has diverse positions; (2) Ring attention with blockwise transformers—combine with memory-efficient attention; (3) DistFlashAttn—integrate with FlashAttention for fused ring implementation. Practical impact: ring attention across 8 GPUs enables 8× context length (e.g., 128K per GPU → 1M total). Used in training long-context models like Gemini (1M+ context). Key enabler for the industry trend toward million-token context windows in production LLMs.