Distributed training coordinates optimization across multiple accelerators and often multiple nodes. Large language, vision, recommendation, scientific, and multimodal models exceed the memory, throughput, or time budget of one device, so useful scale depends on communication-aware partitioning and reliable orchestration. Scaling is not just adding GPUs: global batch, optimizer behavior, numerical reduction order, topology, dataset sharding, checkpoint format, fault policy, and target time-to-quality determine whether more hardware improves the result. A professional system definition specifies the data and model version, numerical precision, batch and sequence shape, parallel topology, storage and network assumptions, target accelerators, failure model, reproducibility boundary, and end-to-end objective. Isolated kernel throughput or one benchmark does not describe delivered training or retrieval behavior.

Architecture, representation, and operating mechanism. Data parallelism replicates the model and shards examples; DDP all-reduces gradients, while FSDP/ZeRO shard parameters, gradients, and optimizer state. Tensor parallelism splits layer operations, pipeline parallelism assigns stages, sequence/context parallelism splits tokens, and expert parallelism distributes MoE experts. Workers load distinct samples, run forward and backward passes, synchronize required tensors, update logically consistent parameters, and advance the data/learning-rate schedule. Collectives include all-reduce, reduce-scatter, all-gather, broadcast, and MoE all-to-all; overlapping communication with compute hides part of their cost. Time to target quality, samples/tokens per second, model FLOP utilization, strong/weak scaling, communication/computation ratio, bubble fraction, straggler tail, HBM peak, host memory, network bytes, checkpoint time, restart time, energy, and cost matter. Accelerators, CPUs, HBM, host RAM, storage, interconnect, schedulers, containers, libraries, compilers, telemetry, registries, APIs, security policy, and operators form one system. Optimizing one stage can move the bottleneck or weaken correctness, isolation, and recoverability. Evaluation reports quality together with throughput, tail latency, accelerator utilization, HBM and host memory, communication volume, storage bandwidth, checkpoint or index cost, energy, fault recovery, scalability, and total cost. Controlled baselines hold data, optimization, hardware, and evaluation constant so an infrastructure change is not confused with extra compute or information.

Implementation, infrastructure, and failure modes. NCCL-class collectives map rings/trees to NVLink, PCIe, InfiniBand, or RoCE; gradient accumulation changes synchronization frequency; mixed precision and loss scaling reduce traffic; activation checkpointing trades compute for memory; fused optimizers, bucketing, prefetch, pinned memory, and topology-aware rank placement improve utilization. Scale-up fabrics connect GPUs within a node or rack, and scale-out fabrics connect nodes. HBM capacity/bandwidth, NIC injection rate, PCIe root placement, GPUDirect/RDMA, switch oversubscription, congestion control, NUMA, storage, power, and cooling constrain delivered training. One slow rank stalls synchronous steps; network congestion creates tail spikes; data duplication or omission changes training; collective mismatch deadlocks; overflow/underflow diverges; pipeline bubbles waste devices; memory fragmentation causes late OOM; checkpoint corruption or rank-topology changes break resume. Engineering includes data movement, finite precision, concurrency, resource contention, security boundaries, error propagation, and deterministic behavior when assumptions fail. Data ingestion, preprocessing, training or indexing, evaluation, artifact registration, deployment, monitoring, refresh, rollback, retention, and deletion form one lifecycle. Dataset, tokenizer, code, dependency, seed, configuration, compiler, kernel, checkpoint, index, prompt, and hardware topology versions remain linked for reproducibility and audit.

Evaluation, governance, and deployment. Compare single- and multi-rank loss trajectories, sample accounting, seed behavior, gradient norms, optimizer states, fault/restart at many points, topology permutations, collective time, overlap, data-loader saturation, communication errors, and scale curves at fixed quality. Schedulers allocate gang resources, containers pin libraries, object storage supplies data, distributed filesystems serve checkpoints, experiment trackers register artifacts, and telemetry correlates job, rank, node, NIC, GPU, and storage behavior. Tenant isolation, dataset access, secret distribution, signed containers, dependency provenance, quota fairness, preemption, retention, cost ownership, incident handling, and reproducible manifests govern shared clusters. Verification combines unit and property tests, numerical references, distributed fault injection, determinism checks, scale tests, performance traces, data-leakage audits, corruption recovery, hardware-in-loop measurement, offline task evaluation, shadow traffic, and canary rollout. Failures are reproducible from immutable artifacts rather than inferred from dashboards. Data ingestion, preprocessing, training or indexing, evaluation, artifact registration, deployment, monitoring, refresh, rollback, retention, and deletion form one lifecycle. Dataset, tokenizer, code, dependency, seed, configuration, compiler, kernel, checkpoint, index, prompt, and hardware topology versions remain linked for reproducibility and audit. Evaluation reports quality together with throughput, tail latency, accelerator utilization, HBM and host memory, communication volume, storage bandwidth, checkpoint or index cost, energy, fault recovery, scalability, and total cost. Controlled baselines hold data, optimization, hardware, and evaluation constant so an infrastructure change is not confused with extra compute or information.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" font-family="-apple-system,Segoe UI,Roboto,sans-serif">
  <rect x="0" y="0" width="760" height="470" fill="#0d1117"/>
  <text x="380" y="28" fill="#e6edf3" font-size="21" font-weight="700" text-anchor="middle">Distributed Training — Parallelism Strategies</text>
  <text x="380" y="48" fill="#8b98a5" font-size="12" text-anchor="middle">data parallel, tensor parallel, pipeline parallel, expert parallel — combine all four to train trillion-param models</text>

  <!-- === TOP: Three parallelism types === -->
  <rect x="25" y="62" width="230" height="148" rx="6" fill="#080d14" stroke="#60a5fa" stroke-width="1.2"/>
  <text x="140" y="82" fill="#93c5fd" font-size="10" text-anchor="middle" font-weight="600">Data Parallel (DP/DDP)</text>
  <text x="45" y="102" fill="#8b98a5" font-size="8.5">Each GPU has full model copy</text>
  <text x="45" y="118" fill="#8b98a5" font-size="8.5">Different data batches per GPU</text>
  <text x="45" y="134" fill="#34d399" font-size="8.5">AllReduce gradients after backward</text>
  <text x="45" y="152" fill="#8b98a5" font-size="8.5">Scales: 8 → 1000s of GPUs</text>
  <text x="45" y="168" fill="#6b7684" font-size="8">PyTorch DDP, FSDP</text>
  <text x="45" y="184" fill="#fbbf24" font-size="8">Memory: full model on each GPU</text>
  <text x="45" y="200" fill="#6b7684" font-size="7.5">Best for: models that fit on 1 GPU</text>

  <rect x="265" y="62" width="230" height="148" rx="6" fill="#080d14" stroke="#34d399" stroke-width="1.2"/>
  <text x="380" y="82" fill="#6ee7b7" font-size="10" text-anchor="middle" font-weight="600">Tensor Parallel (TP)</text>
  <text x="285" y="102" fill="#8b98a5" font-size="8.5">Split weight matrices across GPUs</text>
  <text x="285" y="118" fill="#8b98a5" font-size="8.5">Each GPU computes partial result</text>
  <text x="285" y="134" fill="#34d399" font-size="8.5">AllReduce activations each layer</text>
  <text x="285" y="152" fill="#8b98a5" font-size="8.5">Scales: 2-8 GPUs (within node)</text>
  <text x="285" y="168" fill="#6b7684" font-size="8">Megatron-LM (column/row split)</text>
  <text x="285" y="184" fill="#fbbf24" font-size="8">Memory: 1/N model per GPU</text>
  <text x="285" y="200" fill="#6b7684" font-size="7.5">Best for: large layers (NVLink needed)</text>

  <rect x="505" y="62" width="230" height="148" rx="6" fill="#080d14" stroke="#f59e0b" stroke-width="1.2"/>
  <text x="620" y="82" fill="#fbbf24" font-size="10" text-anchor="middle" font-weight="600">Pipeline Parallel (PP)</text>
  <text x="525" y="102" fill="#8b98a5" font-size="8.5">Split layers across GPUs</text>
  <text x="525" y="118" fill="#8b98a5" font-size="8.5">GPU 0: layers 0-15, GPU 1: 16-31</text>
  <text x="525" y="134" fill="#34d399" font-size="8.5">Micro-batches fill the pipeline</text>
  <text x="525" y="152" fill="#8b98a5" font-size="8.5">Scales: 2-64 GPUs (across nodes)</text>
  <text x="525" y="168" fill="#6b7684" font-size="8">GPipe, PipeDream, interleaved</text>
  <text x="525" y="184" fill="#f87171" font-size="8">Bubble overhead: idle time between</text>
  <text x="525" y="200" fill="#6b7684" font-size="7.5">Best for: very deep models, cross-node</text>

  <!-- === MIDDLE: 3D parallelism === -->
  <rect x="25" y="218" width="710" height="90" rx="6" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="380" y="236" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">3D Parallelism (combine all three) — How GPT-4 / Llama-3 405B trains</text>

  <text x="45" y="258" fill="#8b98a5" font-size="8.5">Example: 512 H100 GPUs training Llama-3 405B</text>
  <text x="45" y="276" fill="#60a5fa" font-size="8.5">TP=8 (within node, NVLink) × PP=8 (across 8 nodes) × DP=8 (8 replicas) = 512 GPUs</text>
  <text x="45" y="294" fill="#34d399" font-size="8.5">+ ZeRO-1 (shard optimizer states across DP ranks) → fits 405B params in memory</text>
  <text x="45" y="308" fill="#6b7684" font-size="8">Communication: TP needs NVLink (900 GB/s), PP needs InfiniBand (400 Gbps), DP uses AllReduce over network</text>

  <!-- === BOTTOM LEFT: DeepSpeed ZeRO === -->
  <rect x="25" y="316" width="350" height="88" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="200" y="334" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">ZeRO (Zero Redundancy Optimizer)</text>

  <text x="45" y="354" fill="#60a5fa" font-size="8.5" font-weight="600">ZeRO-1:</text>
  <text x="105" y="354" fill="#8b98a5" font-size="8.5">shard optimizer states (4× savings)</text>
  <text x="45" y="370" fill="#34d399" font-size="8.5" font-weight="600">ZeRO-2:</text>
  <text x="105" y="370" fill="#8b98a5" font-size="8.5">+ shard gradients (6× savings)</text>
  <text x="45" y="386" fill="#fbbf24" font-size="8.5" font-weight="600">ZeRO-3:</text>
  <text x="105" y="386" fill="#8b98a5" font-size="8.5">+ shard parameters (linear scaling)</text>
  <text x="45" y="400" fill="#6b7684" font-size="8">FSDP (PyTorch) ≈ ZeRO-3 natively</text>

  <!-- === BOTTOM RIGHT: Frameworks === -->
  <rect x="390" y="316" width="345" height="88" rx="5" fill="#0b1220" stroke="#233043" stroke-width="1"/>
  <text x="562" y="334" fill="#e6edf3" font-size="10" text-anchor="middle" font-weight="600">Frameworks</text>

  <text x="410" y="354" fill="#c4b5fd" font-size="8.5" font-weight="600">DeepSpeed (Microsoft):</text>
  <text x="560" y="354" fill="#8b98a5" font-size="8.5">ZeRO, 3D parallel, offload</text>
  <text x="410" y="370" fill="#fbbf24" font-size="8.5" font-weight="600">Megatron-LM (NVIDIA):</text>
  <text x="560" y="370" fill="#8b98a5" font-size="8.5">TP + PP optimized for H100</text>
  <text x="410" y="386" fill="#34d399" font-size="8.5" font-weight="600">PyTorch FSDP:</text>
  <text x="520" y="386" fill="#8b98a5" font-size="8.5">native ZeRO-3, simple API</text>
  <text x="410" y="400" fill="#60a5fa" font-size="8.5" font-weight="600">JAX/XLA (Google):</text>
  <text x="530" y="400" fill="#8b98a5" font-size="8.5">pjit, mesh sharding (TPU pods)</text>

  <!-- Key insight -->
  <rect x="25" y="412" width="710" height="22" rx="3" fill="#0b1220" stroke="#233043" stroke-width="0.8"/>
  <text x="380" y="427" fill="#fbbf24" font-size="9" text-anchor="middle">The art is choosing the right mix: TP for large layers (fast NVLink), PP for depth, DP for scale, ZeRO for memory.</text>

  <text x="380" y="460" fill="#6b7684" font-size="11" text-anchor="middle">Training a frontier model is a distributed systems problem: communication, memory, and compute must all be optimized together.</text>
</svg>

Selection and practical application. Start with data parallelism when the model fits, add state sharding for memory, tensor or pipeline partitioning when layers do not fit, context parallelism for long sequences, and expert parallelism for MoE; map dimensions to the physical topology. Foundation-model pretraining, large-scale fine-tuning, recommender training, distributed vision, protein models, weather models, and scientific surrogate training use hybrid parallelism. Accelerators, CPUs, HBM, host RAM, storage, interconnect, schedulers, containers, libraries, compilers, telemetry, registries, APIs, security policy, and operators form one system. Optimizing one stage can move the bottleneck or weaken correctness, isolation, and recoverability. A professional system definition specifies the data and model version, numerical precision, batch and sequence shape, parallel topology, storage and network assumptions, target accelerators, failure model, reproducibility boundary, and end-to-end objective. Isolated kernel throughput or one benchmark does not describe delivered training or retrieval behavior. CFS connects this topic to semiconductor architecture, implementation, verification, manufacturing, packaging, test, and deployed AI-system tradeoffs across the platform.