tensor parallelism
Tensor parallelism is a technique that splits an individual neural network layer's own weight matrices across multiple devices, letting each device compute part of the same layer's operation in parallel so a model too large for one device can still be trained or run.
```flowchart
{
"rows": [
{ "type": "nodes", "items": [
{ "title": "A layer's full weight matrix is too large for one device", "sub": "single device lacks the memory or compute to hold it alone", "tone": "neutral" }
]},
{ "type": "arrow" },
{ "type": "group", "title": "Weight matrix split across multiple devices", "items": [
{ "title": "Each device computes its own slice of the same layer", "sub": "partial results combined through inter-device communication", "tone": "blue" }
]},
{ "type": "arrow" },
{ "type": "nodes", "items": [
{ "title": "Layer's full output produced as if on one device", "sub": "model scales beyond what any single device could hold", "tone": "green" }
]}
]
}
```
**Tensor parallelism exists because some individual neural network layers in today's largest models are themselves too large, in terms of memory or compute, to fit on a single device, even when the rest of the model is split across devices some other way.** Since a single layer's weight matrix can be mathematically divided into smaller slices without changing the overall computation it represents, tensor parallelism splits that individual layer's own weights across multiple devices, has each device compute its portion of the layer's operation simultaneously, and then combines the partial results, allowing even a single oversized layer to run across hardware that couldn't hold it undivided.
```svg
```
```svg
```
| Aspect | Data parallelism | Tensor parallelism |
|---|---|---|
| What gets split | Training data batches | An individual layer's own weights |
| Solves oversized layers | No, each device needs the full model | Yes |
| Inter-device communication | Mainly gradient synchronization | Frequent, per-layer partial result exchange |
| Common use | Scaling batch size, models that fit per device | Layers too large to fit on a single device |
**Tensor parallelism is typically combined with other parallelism strategies, such as pipeline parallelism, which splits different layers across devices, and data parallelism, which splits training data across device groups, rather than being used entirely on its own.** Because tensor parallelism specifically solves the problem of an individual layer being too large, while other approaches solve different scaling problems, like fitting many layers or processing more training data at once, large-scale training systems commonly combine several parallelism strategies together, each addressing a different dimension of the overall scaling challenge.
**Tensor parallelism requires frequent, fast communication between the devices sharing a split layer, since their partial results must be combined before the next computation step can proceed, making high-bandwidth interconnects between those devices particularly important.** Because devices participating in tensor parallelism need to exchange partial results within a single layer's computation rather than just occasionally synchronizing at the end of a larger step, tensor parallelism places significant demands on interconnect bandwidth and latency between the participating devices, making it most practical among devices connected by especially fast links.
**Tensor parallelism has become an essential technique for both training and serving today's largest language models, since many of their individual layers now exceed the memory capacity of even the most capable single accelerator devices.** Because the largest modern language models contain individual layers whose weight matrices alone can exceed a single accelerator's memory, tensor parallelism has moved from being an optional optimization to a practically necessary technique for both training such models and serving them efficiently at inference time.
Read tensor parallelism through a shared-workbench lens: rather than one worker trying to handle an oversized task alone, several workers each take a section of that same single task and work on their piece simultaneously, combining their partial results into the completed whole.