sequence parallelism training

**Sequence Parallelism** is **the parallelism technique that partitions the sequence dimension across multiple devices to reduce activation memory for long-context training** — enabling training on sequences 4-16× longer than possible on single GPU by distributing activations along sequence length, achieving near-linear scaling when combined with tensor parallelism for models with 32K-100K+ token contexts. **Sequence Parallelism Motivation:** - **Activation Memory Bottleneck**: for sequence length L, batch size B, hidden dimension H, num layers N: activation memory = O(B×L×H×N); grows linearly with sequence length; limits context to 2K-8K tokens on single GPU - **Tensor Parallelism Limitation**: tensor parallelism partitions hidden dimension but not sequence dimension; activations still O(B×L×H/P) per device; sequence length remains bottleneck for long contexts - **Memory Scaling**: doubling sequence length doubles activation memory; 32K context requires 8× memory vs 4K; sequence parallelism enables linear scaling with device count - **Example**: Llama 2 70B with 32K context requires 120GB activation memory; exceeds single A100 80GB; sequence parallelism across 2 GPUs reduces to 60GB per GPU **Sequence Parallelism Strategies:** - **Megatron Sequence Parallelism**: partitions sequence dimension in non-tensor-parallel regions (layer norm, dropout, residual); combined with tensor parallelism for attention/FFN; reduces activation memory by P× where P is tensor parallel size - **Ulysses (All-to-All Sequence Parallelism)**: partitions sequence across devices; uses all-to-all communication to gather full sequence for attention; each device computes attention on full sequence, different heads; enables arbitrary sequence lengths - **Ring Attention**: partitions sequence and KV cache; computes attention in blocks using ring communication; enables training on sequences longer than total GPU memory; extreme memory efficiency - **DeepSpeed-Ulysses**: combines sequence and tensor parallelism; optimizes communication patterns; achieves 2.5× speedup vs Megatron for long sequences; production-ready implementation **Megatron Sequence Parallelism Details:** - **Partitioning Strategy**: partition sequence in layer norm, dropout, residual connections; these operations are sequence-independent; no communication needed during computation - **Communication Points**: all-gather before tensor-parallel regions (attention, FFN); reduce-scatter after tensor-parallel regions; 2 communications per layer; same as tensor parallelism - **Memory Reduction**: reduces activation memory by P× in non-tensor-parallel regions; combined with tensor parallelism, total reduction ~P× for activations; enables P× longer sequences - **Implementation**: requires minimal code changes; integrated in Megatron-LM; automatic when tensor parallelism enabled; transparent to user **Ulysses Sequence Parallelism:** - **All-to-All Communication**: before attention, all-to-all scatter-gather exchanges sequence chunks for head chunks; each device gets full sequence, subset of heads; computes attention independently - **Attention Computation**: each device computes attention for its assigned heads on full sequence; no further communication during attention; results all-to-all gathered after attention - **Communication Volume**: 2 × B × L × H per layer (all-to-all before and after attention); same as tensor parallelism; but enables longer sequences - **Scaling**: near-linear scaling to 8-16 devices; communication overhead 10-20%; enables 8-16× longer sequences; practical for 32K-128K contexts **Ring Attention:** - **Block-Wise Computation**: divides sequence into blocks; each device stores subset of blocks; computes attention using ring communication to access other blocks - **Ring Communication**: devices arranged in ring; pass KV blocks around ring; each device computes attention with local Q and remote KV; accumulates results - **Memory Efficiency**: each device stores only L/P tokens; enables sequences longer than total GPU memory; extreme memory reduction; enables million-token contexts - **Computation Overhead**: each block accessed P times (once per device); P× computation vs standard attention; trade computation for memory; practical for P=4-8 **Performance Characteristics:** - **Memory Reduction**: Megatron SP: P× reduction in non-tensor-parallel activations; Ulysses: enables P× longer sequences; Ring: enables sequences > total memory - **Communication Overhead**: Megatron SP: no additional communication vs tensor parallelism; Ulysses: 2 all-to-all per layer; Ring: ring communication per attention block - **Scaling Efficiency**: Megatron SP: 95%+ efficiency (same as tensor parallelism); Ulysses: 80-90% efficiency; Ring: 50-70% efficiency (high computation overhead) - **Sequence Length**: Megatron SP: 2-4× longer; Ulysses: 8-16× longer; Ring: 100-1000× longer (limited by computation, not memory) **Combining with Other Parallelism:** - **Sequence + Tensor Parallelism**: natural combination; sequence parallelism in non-tensor regions, tensor in attention/FFN; multiplicative memory savings; standard in Megatron-LM - **Sequence + Pipeline Parallelism**: sequence parallelism within pipeline stages; reduces per-stage activation memory; enables longer sequences in pipeline training - **Sequence + Data Parallelism**: replicate sequence-parallel model across data-parallel groups; scales to large clusters; enables large batch sizes on long sequences - **3D + Sequence Parallelism**: combines tensor, pipeline, data, and sequence parallelism; optimal for extreme scale (1000+ GPUs, 100K+ contexts); complex but achieves best efficiency **Use Cases:** - **Long-Context LLMs**: training models with 32K-100K context windows; Llama 2 Long (32K), Code Llama (100K) use sequence parallelism; enables document-level understanding - **Retrieval-Augmented Generation**: processing long retrieved documents; 10K-50K token contexts common; sequence parallelism enables efficient training - **Code Generation**: repository-level code understanding requires 50K-200K tokens; sequence parallelism critical for training on full repositories - **Scientific Text**: processing long papers, books, legal documents; 20K-100K tokens typical; sequence parallelism enables training on full documents **Implementation and Tools:** - **Megatron-LM**: built-in sequence parallelism; automatic when tensor parallelism enabled; production-tested; used for training Llama 2, Code Llama - **DeepSpeed-Ulysses**: Ulysses implementation in DeepSpeed; optimized all-to-all communication; supports hybrid parallelism; easy integration - **Ring Attention**: research implementation available; not yet production-ready; enables extreme sequence lengths; active development - **Framework Support**: PyTorch FSDP exploring sequence parallelism; JAX supports custom parallelism strategies; TensorFlow less mature **Best Practices:** - **Choose Strategy**: Megatron SP for moderate sequences (8K-32K); Ulysses for long sequences (32K-128K); Ring for extreme sequences (>128K) - **Combine with Tensor Parallelism**: always use sequence + tensor parallelism together; multiplicative benefits; standard practice - **Batch Size**: increase batch size with saved memory; improves training stability; typical increase 2-4× vs without sequence parallelism - **Profile Communication**: measure all-to-all overhead; ensure high-bandwidth interconnect (NVLink, InfiniBand); optimize communication patterns Sequence Parallelism is **the technique that breaks the sequence length barrier in transformer training** — by partitioning the sequence dimension across devices, it enables training on contexts 4-16× longer than possible on single GPU, unlocking the long-context capabilities that define the next generation of language models.