sequence parallel

Sequence parallelism splits long sequences across multiple GPUs to enable training and inference with context lengths exceeding single-GPU memory capacity. Ulysses sequence parallelism partitions the sequence dimension across devices, with each GPU processing a subset of tokens. Ring attention extends this by using ring-based communication to pass attention keys and values between GPUs, enabling efficient computation of full attention across the distributed sequence. This approach achieves near-linear scaling with the number of GPUs and enables million-token context windows. Each GPU computes attention for its local query tokens against keys/values from all GPUs, with overlapped communication and computation. Sequence parallelism is orthogonal to tensor and pipeline parallelism, and can be combined for maximum scalability. The technique is critical for long-context applications like document understanding, code generation, and multi-turn conversations. Implementation requires careful optimization of communication patterns and memory management. Ring attention and similar techniques enable context lengths that would be impossible on single devices.