Tensor parallelism is a model parallelism strategy that splits individual weight tensors (matrices) within a layer across multiple devices, enabling each device to compute a portion of every layer's output simultaneously. Unlike pipeline parallelism (which assigns different layers to different devices sequentially), tensor parallelism distributes the computation within each layer, achieving fine-grained parallelism with minimal pipeline idle time (bubble). Tensor parallelism for transformer feedforward layers works by partitioning the weight matrices: the first linear layer's weight matrix W₁ is split column-wise across devices (each device holds a vertical slice), and the second linear layer's weight matrix W₂ is split row-wise (each device holds a horizontal slice). Each device computes its portion of the output independently, and a single all-reduce operation synchronizes the results. For self-attention layers, the query, key, and value projection matrices are split column-wise (each device computes a subset of attention heads), and the output projection is split row-wise — naturally parallelizing multi-head attention. This design, formalized in the Megatron-LM paper by Shoeybi et al. (2019), requires only two all-reduce communication operations per transformer layer (one for the attention block, one for the feedforward block), minimizing communication overhead. Tensor parallelism is most effective within a single machine where devices are connected by high-bandwidth interconnects (NVLink provides 600+ GB/s between GPUs within a node, versus ~25 GB/s for InfiniBand across nodes). Typical configurations use tensor parallelism across 2-8 GPUs within a node and combine it with data parallelism or pipeline parallelism across nodes. Memory savings are proportional to the number of tensor parallel devices — splitting a model across 4 GPUs reduces per-GPU memory by approximately 4×. Tensor parallelism is implemented in Megatron-LM, DeepSpeed, and FairScale, and is essential for training and serving models larger than ~13B parameters.