dynamic loss scaling

**Dynamic loss scaling** is the **adaptive method that adjusts loss scale during mixed-precision training to avoid overflow and underflow** - it automates numeric stabilization for fp16 regimes where gradient magnitude varies over time. **What Is Dynamic loss scaling?** - **Definition**: Multiply loss by a scale factor before backward pass, then unscale gradients before optimizer step. - **Adaptive Logic**: Decrease scale when overflow is detected and increase scale after stable intervals. - **Failure Handling**: Optimizer step can be skipped on overflow to avoid corrupt parameter updates. - **Framework Support**: Implemented in common mixed-precision toolchains for automated stability control. **Why Dynamic loss scaling Matters** - **Numerical Safety**: Protects small gradients from underflow and large gradients from overflow. - **Training Continuity**: Automatic adjustment reduces manual tuning effort across model phases. - **FP16 Viability**: Makes half-precision training practical for a wider range of architectures. - **Operational Robustness**: Adapts to changing gradient distributions during long runs. - **Productivity**: Reduces failed runs caused by precision instability. **How It Is Used in Practice** - **Initial Scale**: Start from a high but safe scale and let runtime controller adjust as needed. - **Overflow Detection**: Check gradients for inf or nan before applying optimizer updates. - **Telemetry**: Log scale value, skipped steps, and overflow events to guide precision debugging. Dynamic loss scaling is **a key stability mechanism for mixed-precision optimization** - adaptive scaling keeps gradients in a representable range while preserving training performance.