data parallel

Data parallelism trains the same model on different data batches across multiple GPUs while model parallelism splits the model itself across GPUs. Hybrid approaches combine both for the largest models. Data parallel is simpler: each GPU has a full model copy processes different batches and synchronizes gradients. This scales linearly until communication overhead dominates. Model parallel splits layers across GPUs necessary when models exceed single GPU memory. Pipeline parallelism divides model into stages processing different batches simultaneously. Tensor parallelism splits individual layers across GPUs. Hybrid parallelism uses data parallel across nodes and model parallel within nodes. ZeRO optimizer reduces memory by partitioning optimizer states gradients and parameters. Frameworks like DeepSpeed Megatron and FSDP implement these strategies. Choosing strategy depends on model size batch size and hardware. Data parallel works for models under 10B parameters. Model parallel is necessary for 100B plus models. Efficient parallelism is essential for training large models enabling models that would not fit on any single GPU.