Model Parallelism Strategies are the techniques for distributing a single neural network across multiple GPUs or nodes when the model is too large to fit on a single device — including tensor parallelism (splitting individual layers), pipeline parallelism (distributing layers across devices), and sequence parallelism (partitioning sequence dimension), enabling training and inference of models with hundreds of billions of parameters.

Tensor Parallelism:

• Layer Splitting: splits weight matrices of individual layers across GPUs; for linear layer Y = XW with W of size [d_in, d_out], split W column-wise across N GPUs; each GPU computes partial output, then all-gather to combine results
• Megatron-LM Approach: splits attention and MLP layers in Transformers; attention: split Q, K, V projections column-wise, output projection row-wise; MLP: split first linear column-wise, second linear row-wise; minimizes communication (2 all-reduce per layer)
• Communication Overhead: requires all-reduce or all-gather after each split layer; communication volume = batch_size × sequence_length × hidden_dim; high-bandwidth interconnect (NVLink, InfiniBand) essential for efficiency
• Scaling Efficiency: near-linear scaling up to 8 GPUs per node (NVLink); efficiency drops with inter-node communication; typically combined with data parallelism for larger scale

Pipeline Parallelism:

• Layer Distribution: assigns consecutive layers to different GPUs; GPU 0: layers 0-7, GPU 1: layers 8-15, etc.; forward pass flows through pipeline, backward pass flows in reverse
• Naive Pipeline Problem: GPU 0 processes batch, sends to GPU 1, then idles while GPU 1 processes; severe underutilization (1/N efficiency for N GPUs)
• Micro-Batching (GPipe): splits batch into micro-batches; GPU 0 processes micro-batch 1, sends to GPU 1, then processes micro-batch 2; overlaps computation across GPUs; achieves ~80-90% efficiency
• Pipeline Bubble: idle time at pipeline start (filling) and end (draining); bubble size = (num_stages - 1) × micro_batch_time; smaller micro-batches reduce bubble but increase communication overhead

Advanced Pipeline Techniques:

• 1F1B (One-Forward-One-Backward): alternates forward and backward micro-batches; reduces memory usage compared to GPipe (stores fewer activations); PipeDream and Megatron use this schedule
• Interleaved Pipeline: each GPU handles multiple non-consecutive stages; GPU 0: layers [0-3, 12-15], GPU 1: layers [4-7, 16-19]; reduces bubble size by enabling more overlapping
• Virtual Pipeline Stages: splits each GPU's layers into multiple virtual stages; increases scheduling flexibility; further reduces bubble at cost of more communication
• Asynchronous Pipeline: doesn't wait for all micro-batches to complete; uses stale gradients for some updates; trades consistency for throughput; requires careful learning rate tuning

Sequence Parallelism:

• Sequence Dimension Splitting: partitions sequence length across GPUs; each GPU processes subset of tokens; used in addition to tensor/pipeline parallelism for very long sequences
• Communication Pattern: requires all-gather for attention (each token attends to all tokens); all-reduce for gradients; communication volume proportional to sequence length
• Megatron Sequence Parallelism: splits sequence dimension for LayerNorm and Dropout (operations outside attention/MLP); reduces activation memory without additional communication
• Ring Attention: processes attention in chunks using ring all-reduce; enables extremely long sequences (millions of tokens); communication overlapped with computation

3D Parallelism:

• Combining Strategies: data parallelism (DP) × tensor parallelism (TP) × pipeline parallelism (PP); example: 1024 GPUs = 8 DP × 8 TP × 16 PP
• Dimension Selection: TP within nodes (high bandwidth), PP across nodes (lower bandwidth), DP for remaining GPUs; matches parallelism strategy to hardware topology
• Megatron-DeepSpeed: combines Megatron's tensor/pipeline parallelism with DeepSpeed's ZeRO optimizer; enables training trillion-parameter models
• Optimal Configuration Search: profile different DP/TP/PP combinations; consider model size, batch size, hardware topology; automated tools (Alpa) search configuration space

Memory Optimization:

• Activation Checkpointing: recomputes activations during backward pass instead of storing; trades computation for memory; enables 2-4× larger models; selective checkpointing (checkpoint every N layers) balances trade-off
• ZeRO (Zero Redundancy Optimizer): partitions optimizer states, gradients, and parameters across data parallel ranks; ZeRO-1 (optimizer states), ZeRO-2 (+gradients), ZeRO-3 (+parameters); reduces memory by DP factor
• Offloading: stores optimizer states or parameters in CPU memory; loads on-demand during computation; ZeRO-Offload, ZeRO-Infinity enable training models larger than total GPU memory
• Mixed Precision: uses FP16/BF16 for activations and gradients, FP32 for optimizer states; reduces memory by 50% for activations; requires loss scaling (FP16) or is numerically stable (BF16)

Communication Optimization:

• Gradient Accumulation: accumulates gradients over multiple micro-batches before communication; reduces communication frequency; effective batch size = micro_batch_size × accumulation_steps × DP_size
• Communication Overlap: overlaps gradient all-reduce with backward computation; starts communication as soon as layer gradients are ready; requires careful scheduling
• Compression: compresses gradients before communication; FP16 instead of FP32 (2× reduction), or quantization to INT8 (4× reduction); trades accuracy for bandwidth
• Hierarchical Communication: all-reduce within nodes (fast NVLink), then across nodes (slower InfiniBand); reduces cross-node traffic; NCCL automatically optimizes communication topology

Framework Support:

• Megatron-LM (NVIDIA): tensor and pipeline parallelism for Transformers; highly optimized for NVIDIA GPUs; used for training GPT, BERT, T5 at scale
• DeepSpeed (Microsoft): ZeRO optimizer, pipeline parallelism, and 3D parallelism; supports PyTorch; extensive optimization for large-scale training
• Alpa (UC Berkeley): automatic parallelization; searches for optimal DP/TP/PP configuration; compiler-based approach; supports JAX
• Fairscale (Meta): modular parallelism components for PyTorch; FSDP (Fully Sharded Data Parallel) similar to ZeRO-3; easier integration than DeepSpeed

Practical Considerations:

• Batch Size Scaling: larger parallelism requires larger batch sizes for efficiency; global_batch_size = micro_batch_size × gradient_accumulation × DP_size; very large batches may hurt convergence
• Learning Rate Tuning: linear scaling rule (LR ∝ batch_size) often works; warmup critical for large batches; may need to tune for specific model/dataset
• Debugging Complexity: distributed training failures are hard to debug; use smaller scale for initial debugging; comprehensive logging and monitoring essential
• Cost-Performance Trade-off: more GPUs = faster training but higher cost; find sweet spot where training time is acceptable and cost is reasonable; consider spot instances for cost savings

Model parallelism strategies are the enabling technology for frontier AI models — without tensor, pipeline, and sequence parallelism, training GPT-4, Llama 3, and other hundred-billion-parameter models would be impossible, making these techniques essential for pushing the boundaries of AI capability.

Distributed AI Training — Scaling from 1 GPU to 100,000. Training frontier LLMs (GPT-4 class, 1–2 trillion parameters) requires distributing computation across thousands of GPUs because no single device has enough memory (80 GB HBM3 holds only 40B parameters in FP16) or compute (1 PFLOPS per GPU vs 10$^{24}$–$10^{25}$ FLOPs total training cost). The four parallelism strategies — data, tensor, pipeline, and expert — partition the workload differently, and production training runs combine all four simultaneously in a 4D parallelism configuration.

FSDP (Fully Sharded Data Parallel) — Memory-Efficient Training. Standard data parallelism replicates the entire model on each GPU — wasteful when models exceed GPU memory. FSDP (PyTorch) and DeepSpeed ZeRO shard model parameters, gradients, and optimizer states across data-parallel ranks. ZeRO Stage 3 reduces per-GPU memory from $16\Psi$ bytes (full replication with Adam FP16) to $16\Psi/N + \text{activations}$. For a 70B model on 64 GPUs: full replication needs 1,120 GB (impossible per GPU); FSDP needs 17.5 GB model memory per GPU + activations — fitting in 80 GB HBM3 with room for large batch sizes. The trade-off: FSDP adds an all-gather before each layer's forward pass and a reduce-scatter after each backward pass, increasing communication volume by 1.5$\times$ versus standard all-reduce.

Model Parallelism — Splitting Layers and Matrices. Tensor parallelism (Megatron-LM) splits the attention and FFN weight matrices column-wise (for the first linear) and row-wise (for the second linear), so each GPU computes a partial result and an all-reduce combines them. For an 8-way TP split: each GPU holds 1/8 of each weight matrix and performs 1/8 of the compute, but requires 2 all-reduce operations per transformer layer (one after attention, one after FFN). At 900 GB/s NVLink bandwidth and 4 ms per all-reduce, TP within a single 8-GPU node adds $<$10% overhead. Pipeline parallelism assigns consecutive layers to different GPUs; the 1F1B (one-forward-one-backward) micro-batch schedule achieves pipeline utilization of $(PP - 1) / PP$ per micro-batch, reaching 90%+ efficiency with 8+ micro-batches per global batch.

Silicon Photonics — Optical I/O for AI. As GPU cluster scale grows from 10,000 to 100,000+ devices, electrical SerDes I/O hits power and reach limits: 112 Gbps PAM4 over copper reaches only 1–2 meters at 10 pJ/bit — insufficient for rack-to-rack communication. Silicon photonics integrates optical modulators, waveguides, and photodetectors on a silicon chip, enabling 1.6 Tbps optical links at 5 pJ/bit over 2+ km of single-mode fiber. Co-packaged optics (CPO) places the photonic engine directly on the switch/GPU package, eliminating pluggable transceiver power overhead. Broadcom, Intel, Marvell, and Ayar Labs ship 800G–1.6T optical engines; next-generation AI clusters (2026+) will use 3.2T CPO to interconnect 100,000 GPUs at $<$1 µs fabric latency.

Transformer Architecture at Hardware Scale. A transformer layer comprises multi-head attention (MHA: $4 d^2$ parameters) and feed-forward network (FFN: $8 d^2$ parameters) for a total of $12 d^2$ parameters per layer. GPT-4 scale ($d = 12{,}288$, 120 layers) yields 1.8T parameters requiring 3.6 TB in FP16 — distributed across 16,000+ GPUs. Training at 55% MFU on 16,384 H100s at 989 TFLOPS FP16 each delivers 8.9 $\times 10^{18}$ FLOPs/s effective; a $10^{25}$ FLOP training run completes in 13 days at 95% uptime. The hardware cost: 16,384 $\times$ 30K USD = 500M USD capital, plus 10–20 MW power at 0.10 USD/kWh = 3–6M USD electricity per run.