Tensor Parallelism and Model Parallelism is distributed training strategies that partition model layers or operations across multiple accelerators — enabling training of models larger than single-device memory through parallel computation of forward and backward passes. Tensor Parallelism and Model Parallelism address the fundamental constraint that modern large language models exceed individual GPU memory capacity. Tensor parallelism partitions weight matrices across devices, with each device computing a subset of output features. For a linear layer with weight matrix W, tensor parallelism splits W row-wise or column-wise across devices. Forward passes require communication to concatenate results from different devices, and backward passes require reduction across devices. This parallelism exposes abundant parallelism — each device performs local computation on partial operations. Model parallelism (pipeline parallelism) divides model layers across devices in sequence. Device 1 processes input through first k layers, passes hidden states to Device 2, which processes through next k layers, and so forth. This creates a pipeline — different devices process different minibatches in flight, improving utilization. Pipeline parallelism reduces per-device memory but requires communication passing large hidden states between devices. Different parallelism strategies have different communication-to-computation ratios. Sequence parallelism partitions sequences across devices, with each device processing a portion of the sequence length. This is particularly valuable for long sequences where sequence length is a primary memory bottleneck. Combined parallelism strategies use tensor parallelism, data parallelism, and pipeline parallelism together. Zero redundancy optimizer (ZeRO) partitions optimizer states, gradients, or parameters across devices, further reducing per-device memory. Flash Attention and other communication-efficient techniques improve parallelism scalability. Ring allreduce and other collective communication patterns optimize communication cost. Network topology and bandwidth significantly impact parallelism efficiency — GPU clusters with high-bandwidth interconnects enable effective scaling to many devices. Load balancing becomes critical in heterogeneous settings — devices with different capability should be utilized proportionally. Gradient accumulation and batch pipelining improve utilization. Research shows that naive model parallelism often performs poorly due to low computation-to-communication ratio, while well-tuned configurations achieve good scaling. Tensor and model parallelism strategies enable distributed training of models exceeding single-device capacity, using different approaches to balance computation and communication across accelerators.