single-node multi-gpu
**Single-node multi-GPU** is the **distributed training configuration where several GPUs in one server collaborate through high-bandwidth local interconnects** - it is often the most efficient starting point for scaling because communication stays inside one machine.
**What Is Single-node multi-GPU?**
- **Definition**: Training setup using all GPUs within one host under one process group or launch context.
- **Communication Path**: Relies on NVLink or PCIe rather than inter-node fabric for gradient exchange.
- **Software Pattern**: Typically implemented with DDP-style data parallelism or local model-parallel groups.
- **Scaling Limit**: Bounded by number of GPUs and memory available in a single server chassis.
**Why Single-node multi-GPU Matters**
- **Low Latency**: Intra-node links are usually faster and more predictable than cross-node networks.
- **Operational Simplicity**: Easier to deploy, debug, and monitor than multi-node distributed clusters.
- **Strong Efficiency**: Often achieves higher scaling efficiency for moderate model sizes.
- **Development Velocity**: Good platform for rapid experimentation before broader cluster rollout.
- **Cost Predictability**: Reduced network complexity lowers operational risk during early scaling stages.
**How It Is Used in Practice**
- **Backend Choice**: Use DDP-style frameworks with NCCL for high-performance local collectives.
- **Rank Affinity**: Bind processes to GPU and NUMA topology for optimal local data paths.
- **Scaling Gate**: Expand to multi-node only after single-node performance is fully optimized.
Single-node multi-GPU training is **the highest-efficiency first step in distributed scaling** - mastering local parallel performance establishes a strong baseline before cross-node complexity is introduced.