megatron-lm

**Megatron-LM** is the **large-model training framework emphasizing tensor parallelism and model-parallel scaling** - it partitions core matrix operations across GPUs to train very large transformer models efficiently. **What Is Megatron-LM?** - **Definition**: NVIDIA framework for training transformer models with combined tensor, pipeline, and data parallelism. - **Tensor Parallel Core**: Splits large matrix multiplications across devices within a node or model-parallel group. - **Communication Need**: Requires high-bandwidth low-latency links due to frequent intra-layer synchronization. - **Scale Target**: Designed for billion- to trillion-parameter language model regimes. **Why Megatron-LM Matters** - **Model Capacity**: Enables architectures too large for single-device memory and compute limits. - **Performance**: Specialized partitioning can improve utilization on dense accelerator systems. - **Research Velocity**: Supports frontier experiments requiring aggressive model scaling. - **Ecosystem Impact**: Influenced many modern LLM training stacks and hybrid parallel designs. - **Hardware Leverage**: Extracts value from NVLink and high-end multi-GPU topology features. **How It Is Used in Practice** - **Parallel Plan**: Choose tensor and pipeline degrees from model shape and network topology. - **Communication Profiling**: Track intra-layer collective overhead to avoid over-partitioning inefficiency. - **Checkpoint Strategy**: Use distributed checkpointing compatible with model-parallel state layout. Megatron-LM is **a foundational framework for tensor-parallel LLM scaling** - effective use depends on careful partition design and communication-aware performance tuning.