tensor parallelism megatron

**Tensor Parallelism** is **the model parallelism technique that partitions individual layers across multiple devices by splitting weight matrices along specific dimensions** — enabling training of models with layers too large for single GPU memory by distributing computation within each layer, achieving near-linear scaling with minimal communication overhead when devices are connected via high-bandwidth interconnects like NVLink. **Tensor Parallelism Fundamentals:** - **Matrix Partitioning**: split weight matrix W ∈ R^(m×n) across P devices; column-wise: each device stores W_i ∈ R^(m×n/P); row-wise: each device stores W_i ∈ R^(m/P×n); reduces memory by P× - **Computation Distribution**: for Y = XW, column partition: each device computes Y_i = XW_i; concatenate results; row partition: each device computes partial Y_i = XW_i; sum results via all-reduce - **Communication Patterns**: column partition requires all-gather after computation; row partition requires all-reduce; communication volume = hidden_size × sequence_length × batch_size; independent of model size - **Transformer Application**: apply to attention (Q, K, V, O projections) and FFN (up, down projections); 6 weight matrices per layer; each partitioned across P devices; reduces per-device memory by P× **Megatron-LM Tensor Parallelism:** - **Attention Partitioning**: split Q, K, V, O matrices column-wise; each device computes subset of attention heads; head_per_device = total_heads / P; independent attention computation; no communication during attention - **FFN Partitioning**: split first linear (up projection) column-wise, second linear (down projection) row-wise; first layer: Y = XW1, each device computes Y_i = XW1_i; second layer: Z = YW2, all-reduce after computation - **Communication Placement**: all-gather after attention output projection; all-reduce after FFN down projection; 2 communications per transformer block; overlapped with computation - **Identity Operators**: insert identity in forward (all-gather/all-reduce), gradient in backward; enables automatic differentiation; elegant implementation; mathematically equivalent to single-device **Memory and Communication:** - **Memory Reduction**: parameters reduced by P×; activations reduced by P× for partitioned dimensions; total memory reduction ~P× for large models; enables models P× larger - **Communication Volume**: 2 × hidden_size × sequence_length × batch_size per layer; independent of model size; scales with sequence length and batch size; not with parameters - **Bandwidth Requirements**: requires high-bandwidth interconnect; NVLink (900 GB/s per GPU) ideal; InfiniBand (200-400 Gb/s) acceptable; Ethernet too slow; intra-node preferred - **Latency Sensitivity**: communication latency critical; sub-microsecond latency needed for efficiency; NVLink provides <1μs; InfiniBand 1-2μs; limits scaling beyond single node **Scaling Efficiency:** - **Intra-Node Scaling**: near-linear scaling within node (2-8 GPUs); NVLink provides sufficient bandwidth; 95-98% efficiency typical; communication fully overlapped with computation - **Inter-Node Scaling**: efficiency degrades with InfiniBand; 80-90% efficiency for 2-4 nodes; 60-80% for 8+ nodes; communication becomes bottleneck; prefer pipeline parallelism for inter-node - **Optimal Parallelism Degree**: P=2-8 for tensor parallelism; beyond 8, communication overhead dominates; combine with pipeline parallelism for larger scale; hybrid approach optimal - **Sequence Length Impact**: longer sequences increase communication volume; reduces efficiency; FlashAttention helps by reducing activation size; critical for long-context models **Implementation Details:** - **Megatron-LM**: NVIDIA's reference implementation; highly optimized; supports tensor, pipeline, data parallelism; used for training GPT-3, Megatron-Turing NLG; production-ready - **Parallelism Mapping**: tensor parallelism within node (NVLink), pipeline across nodes (InfiniBand), data parallelism across pipeline replicas; matches parallelism to hardware topology - **Sequence Parallelism**: extends tensor parallelism to non-partitioned dimensions; reduces activation memory further; enables longer sequences; used in Megatron-LM for extreme contexts - **Selective Activation Recomputation**: recompute activations during backward; reduces memory; combined with tensor parallelism for maximum memory efficiency; enables very large models **Comparison with Pipeline Parallelism:** - **Granularity**: tensor parallelism partitions within layers; pipeline partitions across layers; tensor has finer granularity; better load balance - **Communication**: tensor requires all-gather/all-reduce per layer; pipeline requires point-to-point between stages; tensor needs higher bandwidth; pipeline more flexible - **Efficiency**: tensor achieves 95%+ efficiency with NVLink; pipeline achieves 60-80% with micro-batching; tensor better for intra-node; pipeline better for inter-node - **Memory**: both reduce memory by parallelism degree; tensor reduces per-layer memory; pipeline reduces total model memory; complementary approaches **Advanced Techniques:** - **Sequence Parallelism**: partition sequence dimension in addition to model dimensions; reduces activation memory; enables 2-4× longer sequences; critical for long-context models - **Expert Parallelism**: for Mixture of Experts models, partition experts across devices; combines with tensor parallelism for non-expert layers; enables trillion-parameter MoE models - **Tensor-Pipeline Hybrid**: use tensor parallelism within pipeline stages; reduces per-stage memory; enables larger models; used in Megatron-DeepSpeed for 530B parameters - **Automatic Partitioning**: tools like Alpa automatically determine optimal partitioning strategy; considers hardware topology and model architecture; simplifies deployment **Use Cases:** - **Large Language Models**: GPT-3 175B uses tensor parallelism within nodes; Megatron-Turing 530B uses tensor + pipeline + data; essential for models >10B parameters - **Vision Transformers**: ViT-Huge, ViT-Giant benefit from tensor parallelism; enables training on high-resolution images; reduces per-device memory for large models - **Multi-Modal Models**: CLIP, Flamingo use tensor parallelism for large encoders; enables training on large batch sizes; critical for contrastive learning - **Long-Context Models**: models with 32K-100K context use tensor + sequence parallelism; enables training on long sequences; critical for document understanding **Best Practices:** - **Parallelism Degree**: use P=2-8 for tensor parallelism; match to NVLink topology (8 GPUs per node); beyond 8, use pipeline parallelism; measure efficiency - **Hardware Topology**: use tensor parallelism within NVLink domain; pipeline across InfiniBand; data parallelism for replicas; match parallelism to hardware - **Batch Size**: increase batch size with saved memory; improves efficiency; typical increase 2-8× vs single GPU; balance memory and efficiency - **Profiling**: profile communication and computation; ensure communication overlapped; identify bottlenecks; optimize based on measurements Tensor Parallelism is **the technique that enables training models with layers too large for single GPU** — by partitioning weight matrices and distributing computation within layers, it achieves near-linear scaling on high-bandwidth interconnects, forming the foundation of the parallelism strategies that enable training of the largest language models in existence.