3D-Parallelism & All-Reduce Simulator

Model how an LLM training step splits across a GPU cluster — then run it: the simulation executes on the ChipFoundryServices distributed compute pool. A run is cut along three orthogonal axes whose product must divide the cluster, num_gpus = TP × PP × DP. Tensor parallel (TP) shards each layer’s matmuls and pays four activation all-reduces per layer — bandwidth-hungry, so keep it inside one NVLink node. Pipeline parallel (PP) cuts the layer stack into stages joined by small point-to-point sends, but leaves a fill/drain bubble (PP−1)/(micro-batches+PP−1) of idle time. Data parallel (DP) replicates the model and does one gradient all-reduce per step, usually hidden behind the backward pass. The node rooflines the per-GPU compute (6 × params × tokens FLOPs), adds the exposed TP / PP / DP collectives and the bubble, and returns the step time, the strong-scaling efficiency / model-FLOPs utilisation (MFU), the dominant bottleneck and token throughput — the same TP-vs-PP-vs-DP and interconnect-bandwidth trade-offs that decide how efficiently a Transformer trains on hundreds of accelerators. Reduced-order educational model. See also the power & thermal, training-memory, KV-cache, HBM bandwidth, systolic array, transistor I-V, thermal, interconnect RC, die-yield, 6T SRAM, CMP planarization and lithography simulators and the compute-pool status.

GPT-3 175B · 1024 GPU Llama-3 8B · 8 GPU Comm-bound (thin pipe)
Strong-scaling efficiency vs cluster size — hold TP and PP fixed and grow the data-parallel width (log₂ x-axis). Efficiency (MFU) falls as more GPUs are added because the gradient all-reduce grows while the per-GPU compute shrinks; the dashed line is the 100 % ideal and the marker (●) is your current cluster
Left: efficiency vs interconnect bandwidth — a fatter link hides more communication, so MFU rises toward the 100 % ceiling (dashed). Right: the step-time budget — compute + TP all-reduce + PP point-to-point + pipeline bubble + DP all-reduce as a share of the step. The tallest non-compute bar is your bottleneck
Developer API — same simulation over HTTP (load-balanced across the pool): 
curl -X POST https://www.chipfoundryservices.com/edge/parallelism \
  -H "Content-Type: application/json" \
  -d '{"model_params_b":70,"num_gpus":64,"tensor_parallel":8,"pipeline_parallel":4,
       "micro_batch":1,"gradient_accum_steps":16,"seq_length":4096,"hidden_dim":8192,
       "num_layers":80,"link_bandwidth_gb_s":450,"compute_tflops":500}'
Returns JSON with outputs (data_parallel, used_gpus, idle_gpus, valid_layout, tokens_per_step, compute_time_ms, tp_comm_ms, pp_comm_ms, pipeline_bubble_ms, dp_allreduce_ms, dp_allreduce_raw_ms, step_time_ms, comm_fraction_percent, bubble_fraction_percent, scaling_efficiency_percent, mfu_percent, achieved_tflops_per_gpu, throughput_tokens_s, per_gpu_tokens_s, bottleneck, verdict), the full profile (48-point eff_vs_gpus strong-scaling and eff_vs_bw interconnect sweeps, plus compute_ms, tp_ms, pp_ms, bubble_ms, dp_ms, step_ms, num_gpus, link_bandwidth_gb_s, mfu_percent), the serving node, and compute_ms. Endpoint aliases /edge/parallel, /edge/3d, /edge/tp, /edge/pp, /edge/allreduce.