tensor parallel

Tensor parallelism splits individual model layers across multiple GPUs, distributing matrix operations within a single layer rather than assigning entire layers to different devices, enabling training of models with layers too large for single-GPU memory. Megatron-style parallelism: for transformer attention—split head matrices (Q, K, V projections) column-wise across GPUs; each GPU computes partial attention; reduce at output. For FFN—split first linear column-wise (parallel independent paths), split second linear row-wise (each GPU has part of output), then reduce. Column parallel: W split into [W₁, W₂] columns; each GPU computes W_i × x_i with its portion; no communication until combining. Row parallel: W split into rows; each GPU computes partial output; AllReduce to combine. Communication: tensor parallelism requires AllReduce after each split layer—high communication but within node (NVLink). Optimal use: tensor parallel within node (fast interconnect), pipeline/data parallel across nodes. Combined with sequence parallelism: split activations along sequence dimension during LayerNorm and dropout! Megatron-LM demonstrated: tensor parallelism scales effectively within 8-GPU nodes using NVLink, enabling models with massive individual layers.