tensor parallelism large models

**Tensor Parallelism for Large Models** — Distributing individual tensor operations across multiple devices to train and serve models that exceed single-GPU memory capacity. **Core Partitioning Strategies** — Tensor parallelism splits weight matrices within a single layer across devices, unlike pipeline parallelism which splits layers across stages. Column-parallel partitioning divides weight matrices along the output dimension so each device computes a partial result. Row-parallel partitioning splits along the input dimension, requiring an all-reduce to combine partial sums. Megatron-LM popularized combining column-parallel in the first linear layer with row-parallel in the second, minimizing communication to a single all-reduce per transformer block. **Communication Patterns and Overhead** — The primary communication primitive is all-reduce, which aggregates partial results across tensor-parallel ranks. Communication volume scales with hidden dimension size and batch size. Placing tensor-parallel groups on devices connected via NVLink or NVSwitch minimizes latency compared to cross-node InfiniBand links. Overlapping computation with communication through pipelining partial results reduces idle time on each device. **Implementation Considerations** — Attention heads are naturally parallelizable by assigning subsets of heads to each device. MLP layers require careful partitioning to maintain mathematical equivalence with the sequential version. Dropout and layer normalization must use consistent random seeds or replicated computation across ranks. Activation memory is reduced proportionally to the tensor-parallel degree since each device only stores its partition's activations. **Integration with Other Parallelism Dimensions** — Production systems combine tensor parallelism with data parallelism and pipeline parallelism in 3D parallel configurations. Tensor parallelism typically operates within a single node of 4-8 GPUs while data parallelism spans across nodes. Sequence parallelism extends tensor parallelism by also partitioning layer norm and dropout along the sequence dimension, further reducing memory per device. **Tensor parallelism enables training models with trillions of parameters by distributing computation within layers, making it an essential building block for modern large-scale AI infrastructure.**