spmd programming
**SPMD (Single Program Multiple Data)** is the **dominant parallel programming model where all processors execute the same program but operate on different portions of data, using their processor ID to determine which data to process** — forming the foundation of MPI programming, GPU computing (CUDA), and virtually all large-scale parallel applications, where a single codebase scales from 1 to millions of processors by parameterizing behavior on rank or thread ID rather than writing separate programs for each processor.
**SPMD Concept**
```
Same program, different data:
Rank 0: process(data[0:250]) ← Same code
Rank 1: process(data[250:500]) ← Different data partition
Rank 2: process(data[500:750]) ← Different data partition
Rank 3: process(data[750:1000]) ← Different data partition
```
**SPMD vs. Other Models**
| Model | Description | Example |
|-------|------------|--------|
| SPMD | Same program, different data | MPI, CUDA kernels |
| SIMD | Same instruction, different data | AVX, GPU warp |
| MPMD | Different programs, different data | Client-server, pipeline |
| Master-Worker | One coordinator, many workers | MapReduce |
| BSP | SPMD + supersteps + barriers | Pregel, Apache Giraph |
**MPI SPMD Pattern**
```c
int main(int argc, char **argv) {
MPI_Init(&argc, &argv);
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
// Same code, different behavior based on rank
int chunk = N / size;
int start = rank * chunk;
int end = start + chunk;
// Each rank processes its portion
double local_sum = 0;
for (int i = start; i < end; i++)
local_sum += compute(data[i]);
// Collective: combine results
double global_sum;
MPI_Reduce(&local_sum, &global_sum, 1, MPI_DOUBLE, MPI_SUM, 0, MPI_COMM_WORLD);
MPI_Finalize();
}
```
**CUDA as SPMD**
```cuda
// Every thread runs same kernel, different threadIdx
__global__ void vector_add(float *a, float *b, float *c, int n) {
int id = blockIdx.x * blockDim.x + threadIdx.x; // Unique ID
if (id < n)
c[id] = a[id] + b[id]; // Same operation, different element
}
// Launch: 10000 threads all run vector_add but on different indices
```
**Bulk Synchronous Parallel (BSP)**
```
Superstep 1: [Compute] → [Communicate] → [Barrier]
Superstep 2: [Compute] → [Communicate] → [Barrier]
Superstep 3: [Compute] → [Communicate] → [Barrier]
```
- BSP = SPMD + explicit supersteps.
- Each superstep: Local computation → communication → global barrier.
- Predictable performance: Cost = max(compute) + max(communication) + barrier.
- Used by: Google Pregel (graph processing), Apache Giraph, BSPlib.
**SPMD Advantages**
| Advantage | Why |
|-----------|-----|
| Single codebase | One program maintains, debugs, optimizes |
| Scalable | Same code from 1 to 1M processors |
| Load balanced | Equal data partitions → equal work |
| Portable | MPI SPMD runs on any cluster |
| Composable | Hierarchical SPMD: MPI ranks × OpenMP threads × CUDA blocks |
**SPMD + Data Parallelism in ML**
- Distributed data parallel (DDP): Each GPU runs same model on different mini-batch.
- Same forward pass, same backward pass, different data → classic SPMD.
- AllReduce (gradient sync) = BSP barrier between iterations.
- FSDP: SPMD where each rank holds different model shard.
SPMD is **the programming model that makes large-scale parallelism tractable** — by writing a single program that adapts its behavior based on processor identity, SPMD eliminates the complexity of coordinating different programs while naturally expressing data decomposition, making it the universal foundation that underlies MPI applications on supercomputers, CUDA kernels on GPUs, and distributed training frameworks in machine learning.