Tensor Parallelism is the model parallelism technique that partitions individual layers across multiple devices by splitting weight matrices along specific dimensions — enabling training of models with layers too large for single GPU memory by distributing computation within each layer, achieving near-linear scaling with minimal communication overhead when devices are connected via high-bandwidth interconnects like NVLink.
Tensor Parallelism Fundamentals:
- Matrix Partitioning: split weight matrix W ∈ R^(m×n) across P devices; column-wise: each device stores W_i ∈ R^(m×n/P); row-wise: each device stores W_i ∈ R^(m/P×n); reduces memory by P×
- Computation Distribution: for Y = XW, column partition: each device computes Y_i = XW_i; concatenate results; row partition: each device computes partial Y_i = XW_i; sum results via all-reduce
- Communication Patterns: column partition requires all-gather after computation; row partition requires all-reduce; communication volume = hidden_size × sequence_length × batch_size; independent of model size
- Transformer Application: apply to attention (Q, K, V, O projections) and FFN (up, down projections); 6 weight matrices per layer; each partitioned across P devices; reduces per-device memory by P×
Megatron-LM Tensor Parallelism:
- Attention Partitioning: split Q, K, V, O matrices column-wise; each device computes subset of attention heads; head_per_device = total_heads / P; independent attention computation; no communication during attention
- FFN Partitioning: split first linear (up projection) column-wise, second linear (down projection) row-wise; first layer: Y = XW1, each device computes Y_i = XW1_i; second layer: Z = YW2, all-reduce after computation
- Communication Placement: all-gather after attention output projection; all-reduce after FFN down projection; 2 communications per transformer block; overlapped with computation
- Identity Operators: insert identity in forward (all-gather/all-reduce), gradient in backward; enables automatic differentiation; elegant implementation; mathematically equivalent to single-device
Memory and Communication:
- Memory Reduction: parameters reduced by P×; activations reduced by P× for partitioned dimensions; total memory reduction ~P× for large models; enables models P× larger
- Communication Volume: 2 × hidden_size × sequence_length × batch_size per layer; independent of model size; scales with sequence length and batch size; not with parameters
- Bandwidth Requirements: requires high-bandwidth interconnect; NVLink (900 GB/s per GPU) ideal; InfiniBand (200-400 Gb/s) acceptable; Ethernet too slow; intra-node preferred
- Latency Sensitivity: communication latency critical; sub-microsecond latency needed for efficiency; NVLink provides <1μs; InfiniBand 1-2μs; limits scaling beyond single node
Scaling Efficiency:
- Intra-Node Scaling: near-linear scaling within node (2-8 GPUs); NVLink provides sufficient bandwidth; 95-98% efficiency typical; communication fully overlapped with computation
- Inter-Node Scaling: efficiency degrades with InfiniBand; 80-90% efficiency for 2-4 nodes; 60-80% for 8+ nodes; communication becomes bottleneck; prefer pipeline parallelism for inter-node
- Optimal Parallelism Degree: P=2-8 for tensor parallelism; beyond 8, communication overhead dominates; combine with pipeline parallelism for larger scale; hybrid approach optimal
- Sequence Length Impact: longer sequences increase communication volume; reduces efficiency; FlashAttention helps by reducing activation size; critical for long-context models
Implementation Details:
- Megatron-LM: NVIDIA's reference implementation; highly optimized; supports tensor, pipeline, data parallelism; used for training GPT-3, Megatron-Turing NLG; production-ready
- Parallelism Mapping: tensor parallelism within node (NVLink), pipeline across nodes (InfiniBand), data parallelism across pipeline replicas; matches parallelism to hardware topology
- Sequence Parallelism: extends tensor parallelism to non-partitioned dimensions; reduces activation memory further; enables longer sequences; used in Megatron-LM for extreme contexts
- Selective Activation Recomputation: recompute activations during backward; reduces memory; combined with tensor parallelism for maximum memory efficiency; enables very large models
Comparison with Pipeline Parallelism:
- Granularity: tensor parallelism partitions within layers; pipeline partitions across layers; tensor has finer granularity; better load balance
- Communication: tensor requires all-gather/all-reduce per layer; pipeline requires point-to-point between stages; tensor needs higher bandwidth; pipeline more flexible
- Efficiency: tensor achieves 95%+ efficiency with NVLink; pipeline achieves 60-80% with micro-batching; tensor better for intra-node; pipeline better for inter-node
- Memory: both reduce memory by parallelism degree; tensor reduces per-layer memory; pipeline reduces total model memory; complementary approaches
Advanced Techniques:
- Sequence Parallelism: partition sequence dimension in addition to model dimensions; reduces activation memory; enables 2-4× longer sequences; critical for long-context models
- Expert Parallelism: for Mixture of Experts models, partition experts across devices; combines with tensor parallelism for non-expert layers; enables trillion-parameter MoE models
- Tensor-Pipeline Hybrid: use tensor parallelism within pipeline stages; reduces per-stage memory; enables larger models; used in Megatron-DeepSpeed for 530B parameters
- Automatic Partitioning: tools like Alpa automatically determine optimal partitioning strategy; considers hardware topology and model architecture; simplifies deployment
Use Cases:
- Large Language Models: GPT-3 175B uses tensor parallelism within nodes; Megatron-Turing 530B uses tensor + pipeline + data; essential for models >10B parameters
- Vision Transformers: ViT-Huge, ViT-Giant benefit from tensor parallelism; enables training on high-resolution images; reduces per-device memory for large models
- Multi-Modal Models: CLIP, Flamingo use tensor parallelism for large encoders; enables training on large batch sizes; critical for contrastive learning
- Long-Context Models: models with 32K-100K context use tensor + sequence parallelism; enables training on long sequences; critical for document understanding
Best Practices:
- Parallelism Degree: use P=2-8 for tensor parallelism; match to NVLink topology (8 GPUs per node); beyond 8, use pipeline parallelism; measure efficiency
- Hardware Topology: use tensor parallelism within NVLink domain; pipeline across InfiniBand; data parallelism for replicas; match parallelism to hardware
- Batch Size: increase batch size with saved memory; improves efficiency; typical increase 2-8× vs single GPU; balance memory and efficiency
- Profiling: profile communication and computation; ensure communication overlapped; identify bottlenecks; optimize based on measurements
Tensor Parallelism is the technique that enables training models with layers too large for single GPU — by partitioning weight matrices and distributing computation within layers, it achieves near-linear scaling on high-bandwidth interconnects, forming the foundation of the parallelism strategies that enable training of the largest language models in existence.