parallel computing
**Parallel computing** is the simultaneous use of multiple compute resources — cores, processors, accelerators, or nodes — to solve a problem faster or at larger scale. Every layer of the modern AI-chip stack, from SIMD vector units inside a single core to thousand-GPU training clusters, is a form of parallelism.
```svg
```
| Paradigm | Granularity | Latency | Examples |
|---|---|---|---|
| SIMD / Tensor core | 4–128 lanes | sub-ns | AVX-512, cuTENSOR MMA |
| GPU warp (SIMT) | 32 threads | ns | CUDA, ROCm, PTX |
| Multi-core / SMP | 2–512 cores | ns–µs | OpenMP, pthreads, TBB |
| Multi-GPU (NVLink) | 2–8 GPUs | µs | DDP, tensor parallel |
| GPU cluster (IB) | 100s–1000s | µs–ms | NCCL, MPI, ZeRO |
| Wafer-scale (WSE) | 900K cores | ns on-chip | Cerebras CS-3 |
**Flynn taxonomy** classifies parallel architectures by instruction and data multiplicity. SISD is a serial core. SIMD issues one instruction across N data lanes — the model behind CPU AVX, GPU warp lanes, and systolic-array columns. MIMD allows independent instruction streams on independent data, covering multi-core CPUs and GPU thread blocks simultaneously running different warps. MISD (multiple instruction, single data) is rare, appearing only in fault-tolerant redundant pipelines.
**Amdahl's Law** defines the hard ceiling on speedup: if fraction *s* of a workload is inherently serial, no amount of parallelism can beat 1/s. At 5% serial fraction the ceiling is 20×, no matter how many cores are added. Gustafson's Law offers the optimistic counterpart — as you scale workers, scale the problem size too, and efficiency holds. AI training is close to Gustafson territory: more GPUs let you process bigger batches, not merely the same batch faster.
**GPU SIMT architecture** hides memory latency differently from a CPU: instead of out-of-order execution, an SM switches instantly between warps when one stalls on memory. Occupancy — the ratio of active warps to the maximum the SM can hold — determines how well this latency hiding works. High occupancy requires small register footprint and shared memory usage per thread; this is the central trade-off in GPU kernel optimization.
**Data parallelism in AI training** replicates the full model across GPU ranks, each processing a micro-batch shard. Gradients are synchronized via AllReduce (ring or tree) over NVLink or InfiniBand. ZeRO (Zero Redundancy Optimizer) shards optimizer states, gradients, and parameters across ranks, reducing per-GPU memory by up to 8× at the cost of extra communication. Tensor parallelism (Megatron-LM) partitions weight matrices across ranks — column-parallel for the first linear, row-parallel for the second, with a single AllReduce per transformer layer. Pipeline parallelism splits model layers across GPU ranks, using micro-batch interleaving (GPipe, PipeDream) to fill pipeline bubbles.
**Communication-computation overlap** is the decisive engineering challenge. At scale, all-reduce communication of gradients must be overlapped with backward-pass computation using bucketed gradient synchronization. NVLink bandwidth (900 GB/s on H100, 1800 GB/s on B200) allows tensor-parallel communication within a node with minimal stall; InfiniBand (200 Gb/s) handles inter-node traffic but imposes latency that limits pipeline-parallel throughput.
Read parallel computing through a **decomposition-and-synchronization-cost lens rather than a core-count lens**: the speedup you get depends entirely on how much of the work can be made independent and how cheaply independent pieces can be recombined — not on how many processors you can throw at it.