tensor parallelism distributed llm

**Tensor Parallelism for LLM Training** is a **sophisticated model parallelism approach that partitions weight matrices across multiple GPUs/TPUs, enabling training of trillion-parameter language models by distributing computation and memory load.** **Column and Row Parallel Linear Layers** - **Tensor Parallel Concept**: Weight matrices (W) split across device axis (column or row), enabling parallel matrix multiplication without replicating activations. - **Column-Parallel Linear**: W divided by output dimension (Y = A × W_col, split across GPUs). Each GPU computes partial output; all-reduce aggregates results. - **Row-Parallel Linear**: W divided by input dimension. Each GPU computes partial activation independently; all-gather concatenates results for next layer. - **Mixed Partitioning**: Alternating column→row layers reduces synchronization overhead vs all-column. Megatron-LM uses this pattern for optimal efficiency. **Attention Head Distribution** - **Multi-Head Attention Parallelism**: Attention heads (H heads, typically 96-320) split across tensor-parallel devices. Each device computes subset of attention heads. - **Query/Key/Value Projection Parallelism**: Q/K/V projections use column-parallel layers. Attention computation distributed across heads. - **Attention Dot-Product**: Each device computes (Q × K^T) for its subset of heads independently. Softmax applied per head, values weighted locally. - **Output Projection**: Multi-head outputs concatenated (all-gather), then row-parallel projection aggregates before feeding to MLP. **Megatron-LM 1D/2D/3D Tensor Parallelism** - **1D Tensor Parallelism**: Splits along single dimension (typically embedding or head dimension). Simple implementation but less scalable (synchronization barrier every layer). - **2D Tensor Parallelism**: Creates 2D process grid (N_layer × N_tensor). Reduces all-reduce overhead by pipelining across two dimensions. Megatron-LM sweet spot for 100-500 GPU clusters. - **3D Tensor Parallelism**: Combines tensor parallelism with pipeline and data parallelism. Specialized for extreme scales (>1000 GPUs). Complex scheduling, minimal synchronization overhead. - **Sequence Parallelism Extension**: Splits along sequence dimension (for transformer auto-regressive generation). Reduces attention O(N²) memory complexity. **All-Reduce Communication Patterns** - **All-Reduce Operation**: Collective communication reducing across devices (summation typical in gradient averaging). Each device sends/receives partial results. - **Ring All-Reduce**: Devices arranged in logical ring. Minimizes bandwidth requirement, tolerates network asymmetry. O(NP) communication steps for N data elements, P processes. - **Tree All-Reduce**: Binary tree structure reduces latency to O(log P) hops. Requires bandwidth-saturated links (not always available in over-subscribed networks). - **NCCL (NVIDIA Collective Communications Library)**: Optimized all-reduce kernels, automatically selects best algorithm based on hardware topology and message size. **Activation Memory and Communication Trade-offs** - **Activation Recomputation**: Intermediate activations dropped after forward pass, recomputed during backward pass. Reduces memory by 50% but increases computation 33%. - **Tensor Parallel Memory**: No activation replicas (unlike data parallelism). Memory scales as O(model_size / tensor_parallel_degree + batch_size). - **Communication vs Computation Ratio**: All-reduce bandwidth requirement ~2× (send/receive) weight size per iteration. Optimized via asynchronous communication overlap. - **Network Saturation**: Bandwidth-limited at scales >100 GPUs. Network topology (fat-tree, dragonfly) critical to avoiding communication bottleneck. **Efficiency and Scaling Characteristics** - **Arithmetic Intensity**: Each all-reduce involves O(model_size) bandwidth for O(model_size) computation. Arithmetic intensity ~ 1 FLOP/Byte (memory-bound). - **Scaling Law**: Perfect scaling requires communication hidden behind computation. Overlapping communication with matrix multiplications maintains efficiency to ~64-128 GPU clusters. - **Diminishing Returns**: Beyond tensor_parallel_degree ~64, synchronization overhead dominates. Hybrid 2D/3D parallelism required for 1000+ GPU training. - **Hyperparameter Tuning**: Learning rate, batch size, gradient accumulation adjusted per parallelism configuration. Different configurations yield different convergence behavior.