Sequence Parallelism is the parallelism technique that partitions the sequence dimension across multiple devices to reduce activation memory for long-context training — enabling training on sequences 4-16× longer than possible on single GPU by distributing activations along sequence length, achieving near-linear scaling when combined with tensor parallelism for models with 32K-100K+ token contexts.
Sequence Parallelism Motivation:
- Activation Memory Bottleneck: for sequence length L, batch size B, hidden dimension H, num layers N: activation memory = O(B×L×H×N); grows linearly with sequence length; limits context to 2K-8K tokens on single GPU
- Tensor Parallelism Limitation: tensor parallelism partitions hidden dimension but not sequence dimension; activations still O(B×L×H/P) per device; sequence length remains bottleneck for long contexts
- Memory Scaling: doubling sequence length doubles activation memory; 32K context requires 8× memory vs 4K; sequence parallelism enables linear scaling with device count
- Example: Llama 2 70B with 32K context requires 120GB activation memory; exceeds single A100 80GB; sequence parallelism across 2 GPUs reduces to 60GB per GPU
Sequence Parallelism Strategies:
- Megatron Sequence Parallelism: partitions sequence dimension in non-tensor-parallel regions (layer norm, dropout, residual); combined with tensor parallelism for attention/FFN; reduces activation memory by P× where P is tensor parallel size
- Ulysses (All-to-All Sequence Parallelism): partitions sequence across devices; uses all-to-all communication to gather full sequence for attention; each device computes attention on full sequence, different heads; enables arbitrary sequence lengths
- Ring Attention: partitions sequence and KV cache; computes attention in blocks using ring communication; enables training on sequences longer than total GPU memory; extreme memory efficiency
- DeepSpeed-Ulysses: combines sequence and tensor parallelism; optimizes communication patterns; achieves 2.5× speedup vs Megatron for long sequences; production-ready implementation
Megatron Sequence Parallelism Details:
- Partitioning Strategy: partition sequence in layer norm, dropout, residual connections; these operations are sequence-independent; no communication needed during computation
- Communication Points: all-gather before tensor-parallel regions (attention, FFN); reduce-scatter after tensor-parallel regions; 2 communications per layer; same as tensor parallelism
- Memory Reduction: reduces activation memory by P× in non-tensor-parallel regions; combined with tensor parallelism, total reduction ~P× for activations; enables P× longer sequences
- Implementation: requires minimal code changes; integrated in Megatron-LM; automatic when tensor parallelism enabled; transparent to user
Ulysses Sequence Parallelism:
- All-to-All Communication: before attention, all-to-all scatter-gather exchanges sequence chunks for head chunks; each device gets full sequence, subset of heads; computes attention independently
- Attention Computation: each device computes attention for its assigned heads on full sequence; no further communication during attention; results all-to-all gathered after attention
- Communication Volume: 2 × B × L × H per layer (all-to-all before and after attention); same as tensor parallelism; but enables longer sequences
- Scaling: near-linear scaling to 8-16 devices; communication overhead 10-20%; enables 8-16× longer sequences; practical for 32K-128K contexts
Ring Attention:
- Block-Wise Computation: divides sequence into blocks; each device stores subset of blocks; computes attention using ring communication to access other blocks
- Ring Communication: devices arranged in ring; pass KV blocks around ring; each device computes attention with local Q and remote KV; accumulates results
- Memory Efficiency: each device stores only L/P tokens; enables sequences longer than total GPU memory; extreme memory reduction; enables million-token contexts
- Computation Overhead: each block accessed P times (once per device); P× computation vs standard attention; trade computation for memory; practical for P=4-8
Performance Characteristics:
- Memory Reduction: Megatron SP: P× reduction in non-tensor-parallel activations; Ulysses: enables P× longer sequences; Ring: enables sequences > total memory
- Communication Overhead: Megatron SP: no additional communication vs tensor parallelism; Ulysses: 2 all-to-all per layer; Ring: ring communication per attention block
- Scaling Efficiency: Megatron SP: 95%+ efficiency (same as tensor parallelism); Ulysses: 80-90% efficiency; Ring: 50-70% efficiency (high computation overhead)
- Sequence Length: Megatron SP: 2-4× longer; Ulysses: 8-16× longer; Ring: 100-1000× longer (limited by computation, not memory)
Combining with Other Parallelism:
- Sequence + Tensor Parallelism: natural combination; sequence parallelism in non-tensor regions, tensor in attention/FFN; multiplicative memory savings; standard in Megatron-LM
- Sequence + Pipeline Parallelism: sequence parallelism within pipeline stages; reduces per-stage activation memory; enables longer sequences in pipeline training
- Sequence + Data Parallelism: replicate sequence-parallel model across data-parallel groups; scales to large clusters; enables large batch sizes on long sequences
- 3D + Sequence Parallelism: combines tensor, pipeline, data, and sequence parallelism; optimal for extreme scale (1000+ GPUs, 100K+ contexts); complex but achieves best efficiency
Use Cases:
- Long-Context LLMs: training models with 32K-100K context windows; Llama 2 Long (32K), Code Llama (100K) use sequence parallelism; enables document-level understanding
- Retrieval-Augmented Generation: processing long retrieved documents; 10K-50K token contexts common; sequence parallelism enables efficient training
- Code Generation: repository-level code understanding requires 50K-200K tokens; sequence parallelism critical for training on full repositories
- Scientific Text: processing long papers, books, legal documents; 20K-100K tokens typical; sequence parallelism enables training on full documents
Implementation and Tools:
- Megatron-LM: built-in sequence parallelism; automatic when tensor parallelism enabled; production-tested; used for training Llama 2, Code Llama
- DeepSpeed-Ulysses: Ulysses implementation in DeepSpeed; optimized all-to-all communication; supports hybrid parallelism; easy integration
- Ring Attention: research implementation available; not yet production-ready; enables extreme sequence lengths; active development
- Framework Support: PyTorch FSDP exploring sequence parallelism; JAX supports custom parallelism strategies; TensorFlow less mature
Best Practices:
- Choose Strategy: Megatron SP for moderate sequences (8K-32K); Ulysses for long sequences (32K-128K); Ring for extreme sequences (>128K)
- Combine with Tensor Parallelism: always use sequence + tensor parallelism together; multiplicative benefits; standard practice
- Batch Size: increase batch size with saved memory; improves training stability; typical increase 2-4× vs without sequence parallelism
- Profile Communication: measure all-to-all overhead; ensure high-bandwidth interconnect (NVLink, InfiniBand); optimize communication patterns
Sequence Parallelism is the technique that breaks the sequence length barrier in transformer training — by partitioning the sequence dimension across devices, it enables training on contexts 4-16× longer than possible on single GPU, unlocking the long-context capabilities that define the next generation of language models.