Parallel computing is the simultaneous use of multiple compute resources — cores, processors, accelerators, or nodes — to solve a problem faster or at larger scale. Every layer of the modern AI-chip stack, from SIMD vector units inside a single core to thousand-GPU training clusters, is a form of parallelism.

<svg viewBox="0 0 760 470" xmlns="http://www.w3.org/2000/svg" role="img" aria-label="Parallel Computing — Flynn taxonomy, thread/process model, GPU SIMT and scaling laws">
<defs>
<style>text{font-family:ui-monospace,'Cascadia Code',SFMono-Regular,Consolas,monospace;fill:#c9d1d9}.dim{fill:#8b949e}.hdr{font-size:12px;font-weight:700;fill:#e6edf3}.lrg{font-size:11px;fill:#e6edf3;font-weight:700}.med{font-size:10px}.sm{font-size:9px}</style>
<marker id="pc_arr" markerWidth="7" markerHeight="7" refX="3" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6" fill="#8b949e"/></marker>
<marker id="pc_garr" markerWidth="7" markerHeight="7" refX="3" refY="3" orient="auto"><path d="M0,0 L6,3 L0,6" fill="#3fb950"/></marker>
</defs>
<rect width="760" height="470" fill="#0d1117"/>

<!-- ═══ PANEL 1: Flynn Taxonomy (x=8,y=8,w=234,h=290) ═══ -->
<rect x="8" y="8" width="234" height="290" rx="6" fill="#161b22"/>
<text x="125" y="27" text-anchor="middle" class="hdr">Flynn Taxonomy</text>

<!-- 2×2 grid: SISD, SIMD, MISD, MIMD -->
<!-- Labels -->
<text x="90" y="46" text-anchor="middle" class="sm dim">Single Data</text>
<text x="180" y="46" text-anchor="middle" class="sm dim">Multiple Data</text>
<text x="28" y="90" class="sm dim" transform="rotate(-90 28 90)">Single Instr</text>
<text x="28" y="190" class="sm dim" transform="rotate(-90 28 190)">Multiple Instr</text>

<!-- SISD box -->
<rect x="44" y="52" width="90" height="70" rx="4" fill="#21262d" stroke="#30363d" stroke-width="1.2"/>
<text x="89" y="74" text-anchor="middle" class="med" font-weight="700" fill="#c9d1d9">SISD</text>
<text x="89" y="90" text-anchor="middle" class="sm dim">1 instr</text>
<text x="89" y="103" text-anchor="middle" class="sm dim">1 data stream</text>
<text x="89" y="116" text-anchor="middle" class="sm dim">serial CPU core</text>

<!-- SIMD box — highlighted (GPU relevant) -->
<rect x="140" y="52" width="96" height="70" rx="4" fill="#0d419d" stroke="#388bfd" stroke-width="1.5"/>
<text x="188" y="74" text-anchor="middle" class="med" font-weight="700" fill="#a5d6ff">SIMD</text>
<text x="188" y="90" text-anchor="middle" class="sm" fill="#a5d6ff">1 instr</text>
<text x="188" y="103" text-anchor="middle" class="sm" fill="#a5d6ff">N data streams</text>
<text x="188" y="116" text-anchor="middle" class="sm" fill="#a5d6ff">AVX / GPU lane</text>

<!-- MISD box -->
<rect x="44" y="132" width="90" height="70" rx="4" fill="#21262d" stroke="#30363d" stroke-width="1.2"/>
<text x="89" y="154" text-anchor="middle" class="med" font-weight="700" fill="#8b949e">MISD</text>
<text x="89" y="170" text-anchor="middle" class="sm dim">rare in practice</text>
<text x="89" y="183" text-anchor="middle" class="sm dim">fault-tolerant</text>
<text x="89" y="196" text-anchor="middle" class="sm dim">pipelines only</text>

<!-- MIMD box — highlighted (multi-core, distributed) -->
<rect x="140" y="132" width="96" height="70" rx="4" fill="#0f2d12" stroke="#3fb950" stroke-width="1.5"/>
<text x="188" y="154" text-anchor="middle" class="med" font-weight="700" fill="#3fb950">MIMD</text>
<text x="188" y="170" text-anchor="middle" class="sm" fill="#3fb950">N instrs</text>
<text x="188" y="183" text-anchor="middle" class="sm" fill="#3fb950">N data streams</text>
<text x="188" y="196" text-anchor="middle" class="sm" fill="#3fb950">multi-core, cluster</text>

<!-- Dividers -->
<line x1="137" y1="52" x2="137" y2="202" stroke="#30363d" stroke-width="1"/>
<line x1="44" y1="129" x2="236" y2="129" stroke="#30363d" stroke-width="1"/>

<!-- Amdahl's Law -->
<rect x="16" y="212" width="218" height="78" rx="4" fill="#0d1117" stroke="#30363d" stroke-width="1"/>
<text x="125" y="228" text-anchor="middle" class="lrg">Amdahl&#x2019;s Law</text>
<text x="125" y="244" text-anchor="middle" class="med" fill="#d29922">Speedup &#8804; 1 / (s + (1&#8722;s)/N)</text>
<text x="22" y="260" class="sm dim">s = serial fraction, N = processors</text>
<text x="22" y="274" class="sm" fill="#f85149">s=0.05 &#8594; max speedup &#8776;20&#215; (even &#8734; cores)</text>
<text x="22" y="284" class="sm dim">Gustafson: scale problem with N &#8594; linear gains</text>


<!-- ═══ PANEL 2: Thread / Process Model (x=250,y=8,w=244,h=290) ═══ -->
<rect x="250" y="8" width="244" height="290" rx="6" fill="#161b22"/>
<text x="372" y="27" text-anchor="middle" class="hdr">Thread &#183; Process &#183; GPU SIMT</text>

<!-- Process box -->
<rect x="260" y="38" width="224" height="52" rx="4" fill="#0d1117" stroke="#388bfd" stroke-width="1.2"/>
<text x="270" y="55" class="sm" fill="#388bfd" font-weight="700">Process (heavyweight)</text>
<text x="270" y="68" class="sm dim">own address space &#183; OS-scheduled &#183; IPC via pipes/sockets/SHM</text>
<text x="270" y="80" class="sm dim">context switch ~1&#x2013;10 &#xb5;s &#183; multiprocessing.Pool, MPI ranks</text>

<!-- Thread box -->
<rect x="260" y="98" width="224" height="52" rx="4" fill="#0d1117" stroke="#3fb950" stroke-width="1.2"/>
<text x="270" y="115" class="sm" fill="#3fb950" font-weight="700">Thread (lightweight)</text>
<text x="270" y="128" class="sm dim">shared heap &#183; OS/user scheduled &#183; sync via mutex/semaphore</text>
<text x="270" y="140" class="sm dim">context switch ~0.1&#x2013;1 &#xb5;s &#183; pthreads, OpenMP, std::thread</text>

<!-- GPU SIMT block -->
<rect x="260" y="158" width="224" height="132" rx="4" fill="#0d419d" stroke="#388bfd" stroke-width="1.5"/>
<text x="270" y="175" class="sm" fill="#a5d6ff" font-weight="700">GPU SIMT — 1000s of threads</text>

<!-- Warp diagram: 32 tiny thread squares -->
<text x="270" y="192" class="sm dim">1 warp = 32 threads, same instruction</text>

<!-- Row of 16 thread squares (2 rows of 16 = 32 threads) -->
<g fill="#388bfd" opacity="0.8">
<rect x="270" y="196" width="7" height="7" rx="1"/><rect x="280" y="196" width="7" height="7" rx="1"/>
<rect x="290" y="196" width="7" height="7" rx="1"/><rect x="300" y="196" width="7" height="7" rx="1"/>
<rect x="310" y="196" width="7" height="7" rx="1"/><rect x="320" y="196" width="7" height="7" rx="1"/>
<rect x="330" y="196" width="7" height="7" rx="1"/><rect x="340" y="196" width="7" height="7" rx="1"/>
<rect x="350" y="196" width="7" height="7" rx="1"/><rect x="360" y="196" width="7" height="7" rx="1"/>
<rect x="370" y="196" width="7" height="7" rx="1"/><rect x="380" y="196" width="7" height="7" rx="1"/>
<rect x="390" y="196" width="7" height="7" rx="1"/><rect x="400" y="196" width="7" height="7" rx="1"/>
<rect x="410" y="196" width="7" height="7" rx="1"/><rect x="420" y="196" width="7" height="7" rx="1"/>
</g>
<g fill="#388bfd" opacity="0.8">
<rect x="270" y="206" width="7" height="7" rx="1"/><rect x="280" y="206" width="7" height="7" rx="1"/>
<rect x="290" y="206" width="7" height="7" rx="1"/><rect x="300" y="206" width="7" height="7" rx="1"/>
<rect x="310" y="206" width="7" height="7" rx="1"/><rect x="320" y="206" width="7" height="7" rx="1"/>
<rect x="330" y="206" width="7" height="7" rx="1"/><rect x="340" y="206" width="7" height="7" rx="1"/>
<rect x="350" y="206" width="7" height="7" rx="1"/><rect x="360" y="206" width="7" height="7" rx="1"/>
<rect x="370" y="206" width="7" height="7" rx="1"/><rect x="380" y="206" width="7" height="7" rx="1"/>
<rect x="390" y="206" width="7" height="7" rx="1"/><rect x="400" y="206" width="7" height="7" rx="1"/>
<rect x="410" y="206" width="7" height="7" rx="1"/><rect x="420" y="206" width="7" height="7" rx="1"/>
</g>
<text x="432" y="205" class="sm" fill="#a5d6ff">&#215;32</text>

<text x="270" y="226" class="sm dim">divergence: branches mask lanes (&#8593; waste)</text>
<text x="270" y="240" class="sm dim">occupancy: active warps / max warps per SM</text>
<text x="270" y="254" class="sm" fill="#3fb950">H100: 132 SMs &#215; 64 warps &#215; 32 = 270,336 threads</text>
<text x="270" y="268" class="sm dim">hide latency via warp switching, not clock speed</text>
<text x="270" y="282" class="sm dim">shared mem = L1 scratchpad per SM block</text>


<!-- ═══ PANEL 3: Parallelism Patterns (x=502,y=8,w=250,h=290) ═══ -->
<rect x="502" y="8" width="250" height="290" rx="6" fill="#161b22"/>
<text x="627" y="27" text-anchor="middle" class="hdr">Parallelism Patterns</text>

<!-- Data Parallelism -->
<rect x="512" y="36" width="230" height="56" rx="4" fill="#0d1117" stroke="#3fb950" stroke-width="1.2"/>
<text x="520" y="52" class="sm" fill="#3fb950" font-weight="700">Data Parallelism</text>
<text x="520" y="66" class="sm dim">same op on different data shards</text>
<text x="520" y="78" class="sm dim">SIMD lanes, GPU warps, PyTorch DDP, MPI allreduce</text>
<text x="520" y="88" class="sm dim">strong scaling: fixed problem, add workers</text>

<!-- Task Parallelism -->
<rect x="512" y="100" width="230" height="56" rx="4" fill="#0d1117" stroke="#d29922" stroke-width="1.2"/>
<text x="520" y="116" class="sm" fill="#d29922" font-weight="700">Task Parallelism</text>
<text x="520" y="130" class="sm dim">different tasks on different processors</text>
<text x="520" y="142" class="sm dim">pipeline stages, DAG workers, async I/O</text>
<text x="520" y="152" class="sm dim">weak scaling: bigger problem, add workers</text>

<!-- Pipeline Parallelism -->
<rect x="512" y="164" width="230" height="56" rx="4" fill="#0d1117" stroke="#388bfd" stroke-width="1.2"/>
<text x="520" y="180" class="sm" fill="#388bfd" font-weight="700">Pipeline Parallelism (AI)</text>
<text x="520" y="194" class="sm dim">model layers split across GPU ranks</text>

<!-- Pipeline stage boxes -->
<rect x="520" y="198" width="38" height="14" rx="2" fill="#0d419d" stroke="#388bfd" stroke-width="1"/>
<text x="539" y="209" text-anchor="middle" class="sm" fill="#a5d6ff">L1&#8211;4</text>
<line x1="558" y1="205" x2="565" y2="205" stroke="#388bfd" marker-end="url(#pc_arr)"/>
<rect x="566" y="198" width="38" height="14" rx="2" fill="#0d419d" stroke="#388bfd" stroke-width="1"/>
<text x="585" y="209" text-anchor="middle" class="sm" fill="#a5d6ff">L5&#8211;8</text>
<line x1="604" y1="205" x2="611" y2="205" stroke="#388bfd" marker-end="url(#pc_arr)"/>
<rect x="612" y="198" width="38" height="14" rx="2" fill="#0d419d" stroke="#388bfd" stroke-width="1"/>
<text x="631" y="209" text-anchor="middle" class="sm" fill="#a5d6ff">L9&#8211;12</text>
<line x1="650" y1="205" x2="657" y2="205" stroke="#388bfd" marker-end="url(#pc_arr)"/>
<rect x="658" y="198" width="38" height="14" rx="2" fill="#0d419d" stroke="#388bfd" stroke-width="1"/>
<text x="677" y="209" text-anchor="middle" class="sm" fill="#a5d6ff">L13&#8211;N</text>

<text x="520" y="218" class="sm dim">bubble fill &#8594; micro-batch interleaving (GPipe)</text>

<!-- Tensor / Model Parallelism -->
<rect x="512" y="228" width="230" height="62" rx="4" fill="#0d1117" stroke="#f85149" stroke-width="1.2"/>
<text x="520" y="244" class="sm" fill="#f85149" font-weight="700">Tensor Parallelism (Megatron)</text>
<text x="520" y="258" class="sm dim">split weight matrix across GPU ranks</text>
<text x="520" y="270" class="sm dim">col-split A &#8594; row-split B &#8594; allreduce result</text>
<text x="520" y="282" class="sm dim">needs NVLink for all-reduce bandwidth</text>


<!-- ═══ CARDS ROW (y=305,h=158) ═══ -->
<!-- Card 1 -->
<rect x="8" y="305" width="234" height="158" rx="6" fill="#161b22"/>
<text x="125" y="322" text-anchor="middle" class="lrg">Synchronization</text>
<text x="16" y="338" class="sm" fill="#f85149">Race condition: unsynchronized shared write</text>
<text x="16" y="352" class="sm dim">Mutex: mutual exclusion, one thread at a time</text>
<text x="16" y="366" class="sm dim">Semaphore: counting gate for N resources</text>
<text x="16" y="380" class="sm dim">Barrier: all threads wait before proceeding</text>
<text x="16" y="394" class="sm dim">Atomic op: read-modify-write in one instruction</text>
<text x="16" y="408" class="sm" fill="#3fb950">Lock-free: CAS loops, avoid OS overhead</text>
<text x="16" y="422" class="sm dim">False sharing: cache-line ping-pong (avoid)</text>
<text x="16" y="436" class="sm dim">Memory model: happens-before ordering rules</text>
<text x="16" y="450" class="sm dim">CUDA: __syncthreads() within a block only</text>

<!-- Card 2 -->
<rect x="250" y="305" width="244" height="158" rx="6" fill="#161b22"/>
<text x="372" y="322" text-anchor="middle" class="lrg">Interconnects for Parallelism</text>
<text x="258" y="338" class="sm dim">Scale-up: NVLink 4 &#8594; 900 GB/s bidirectional</text>
<text x="258" y="352" class="sm" fill="#a5d6ff">NVLink 5 (B200): 1800 GB/s bidi per GPU</text>
<text x="258" y="366" class="sm dim">Scale-out: InfiniBand HDR 200 Gb/s per port</text>
<text x="258" y="380" class="sm dim">Ethernet 400GbE for commodity GPU clusters</text>
<text x="258" y="394" class="sm dim">AllReduce: ring (bandwidth-optimal) or tree</text>
<text x="258" y="408" class="sm dim">NCCL: NVIDIA collective comms library</text>
<text x="258" y="422" class="sm dim">Communication-computation overlap is critical</text>
<text x="258" y="436" class="sm" fill="#3fb950">ZeRO: shard optimizer state to cut memory</text>
<text x="258" y="450" class="sm dim">Reduce bus contention: bucket gradients</text>

<!-- Card 3 -->
<rect x="502" y="305" width="250" height="158" rx="6" fill="#161b22"/>
<text x="627" y="322" text-anchor="middle" class="lrg">AI Chip Parallelism Stack</text>
<text x="510" y="338" class="sm dim">Instruction-level: OOO, superscalar within core</text>
<text x="510" y="352" class="sm" fill="#a5d6ff">SIMD / tensor core: FP16/INT8 matrix tiles</text>
<text x="510" y="366" class="sm" fill="#a5d6ff">Thread-level: warps on SM, latency hiding</text>
<text x="510" y="380" class="sm dim">Block-level: CTA across SMs, shared mem</text>
<text x="510" y="394" class="sm dim">Data parallel: DDP, replicate model N GPUs</text>
<text x="510" y="408" class="sm dim">Tensor parallel: column/row split (Megatron)</text>
<text x="510" y="422" class="sm dim">Pipeline parallel: stages across GPU ranks</text>
<text x="510" y="436" class="sm dim">Expert parallel: MoE routing across GPUs</text>
<text x="510" y="450" class="sm dim">3D parallel: DP &#215; TP &#215; PP combined</text>
</svg>

Flynn taxonomy classifies parallel architectures by instruction and data multiplicity. SISD is a serial core. SIMD issues one instruction across N data lanes — the model behind CPU AVX, GPU warp lanes, and systolic-array columns. MIMD allows independent instruction streams on independent data, covering multi-core CPUs and GPU thread blocks simultaneously running different warps. MISD (multiple instruction, single data) is rare, appearing only in fault-tolerant redundant pipelines.

Amdahl's Law defines the hard ceiling on speedup: if fraction s of a workload is inherently serial, no amount of parallelism can beat 1/s. At 5% serial fraction the ceiling is 20×, no matter how many cores are added. Gustafson's Law offers the optimistic counterpart — as you scale workers, scale the problem size too, and efficiency holds. AI training is close to Gustafson territory: more GPUs let you process bigger batches, not merely the same batch faster.

GPU SIMT architecture hides memory latency differently from a CPU: instead of out-of-order execution, an SM switches instantly between warps when one stalls on memory. Occupancy — the ratio of active warps to the maximum the SM can hold — determines how well this latency hiding works. High occupancy requires small register footprint and shared memory usage per thread; this is the central trade-off in GPU kernel optimization.

Data parallelism in AI training replicates the full model across GPU ranks, each processing a micro-batch shard. Gradients are synchronized via AllReduce (ring or tree) over NVLink or InfiniBand. ZeRO (Zero Redundancy Optimizer) shards optimizer states, gradients, and parameters across ranks, reducing per-GPU memory by up to 8× at the cost of extra communication. Tensor parallelism (Megatron-LM) partitions weight matrices across ranks — column-parallel for the first linear, row-parallel for the second, with a single AllReduce per transformer layer. Pipeline parallelism splits model layers across GPU ranks, using micro-batch interleaving (GPipe, PipeDream) to fill pipeline bubbles.

Communication-computation overlap is the decisive engineering challenge. At scale, all-reduce communication of gradients must be overlapped with backward-pass computation using bucketed gradient synchronization. NVLink bandwidth (900 GB/s on H100, 1800 GB/s on B200) allows tensor-parallel communication within a node with minimal stall; InfiniBand (200 Gb/s) handles inter-node traffic but imposes latency that limits pipeline-parallel throughput.

Read parallel computing through a decomposition-and-synchronization-cost lens rather than a core-count lens: the speedup you get depends entirely on how much of the work can be made independent and how cheaply independent pieces can be recombined — not on how many processors you can throw at it.