model parallelism vs data parallelism

**Data parallelism and model parallelism are the two fundamentally different ways to split a training job across many chips, and they solve two completely different problems — one deals with too much data, the other with too much model.** The all-reduce entry described how chips combine their gradients after each computes an update, but it didn't say what each chip was actually computing on in the first place, and that's exactly where these two approaches diverge. Data parallelism keeps a full copy of the entire model on every single chip and gives each chip a different slice of the training data to process; model parallelism does the opposite, splitting the model itself into pieces spread across chips, so no single chip ever holds the whole thing at once. Real large-scale training almost always uses both simultaneously, but understanding them separately is what makes combining them make sense. **Data parallelism is the simpler idea and the default choice whenever the model actually fits on one chip.** Every chip runs an identical copy of the full model, but each one processes a different batch of training examples in parallel, computing its own gradient from its own slice of data. Those independently-computed gradients are exactly what all-reduce then combines into one shared update, applied identically to every chip's copy of the model, keeping all copies in perfect sync step after step. This scales training speed almost linearly with chip count for as long as the model comfortably fits in a single chip's memory — the limiting factor isn't compute distribution, it's simply how much data can be pushed through the cluster per unit time. ```svg ``` **Model parallelism becomes necessary once a model simply cannot fit on a single chip's memory at all, no matter how the data is split.** The largest current models have parameter counts that outgrow even the most memory-rich single accelerator, so the model itself has to be partitioned — commonly by splitting it layer-by-layer across chips (pipeline parallelism) or by splitting individual large operations, like the enormous matrix multiplies inside each layer, across chips (tensor parallelism). Either way, a single forward pass now requires chips to pass intermediate results to each other mid-computation, since no one chip has everything needed to finish the pass alone. This introduces a communication cost data parallelism never has to pay: model-parallel chips must talk to each other constantly, throughout every single forward and backward pass, rather than just once per step, which is why model parallelism demands an even faster, lower-latency interconnect between chips than data parallelism's periodic all-reduce. | Approach | What's Split | Communication Pattern | Needed When | |---|---|---|---| | Data Parallelism | Training data, across identical model copies | Gradients combined once per step (all-reduce) | Model fits comfortably on one chip | | Pipeline Parallelism (model) | Model layers, across chips in sequence | Activations passed chip-to-chip every layer boundary | Model too large for one chip's memory | | Tensor Parallelism (model) | Individual large operations within a layer | Partial results exchanged within every single operation | Even one layer is too large for one chip | ```flowchart st=>start: Training job needs to run across a cluster of chips check=>operation: Does the full model fit comfortably in one chip's memory? dp=>operation: If yes, use data parallelism — full model copy per chip, split the data mp=>operation: If no, use model parallelism — split the model itself across chips combine=>operation: Real large-scale training typically combines both simultaneously allreduce=>operation: Data-parallel gradients synchronized via all-reduce between parallel model copies pipeline=>operation: Model-parallel chips exchange activations mid-pass within each copy pass=>end: Cluster trains a model too large or too data-hungry for any single chip alone st->check->dp->mp->combine->allreduce->pipeline->pass ``` **In practice, the largest training runs use both approaches layered on top of each other, and this is exactly why training-cluster interconnect design, covered in the all-reduce entry, has to handle two very different communication patterns at once.** A cluster might be organized into groups of chips that each hold one full model-parallel "slice" via pipeline and tensor parallelism, while multiple such groups run in data parallelism alongside each other, each processing different data and periodically all-reducing their gradients across groups. This layered structure is why modern AI accelerator clusters need both very fast, low-latency links within a tightly-coupled group of chips (for model parallelism's constant activation-passing) and efficient, high-bandwidth links between larger groups (for data parallelism's periodic gradient synchronization) — two distinct interconnect requirements, both essential, both traced directly back to the two different splitting strategies described here.