mixed precision training fp16 bf16

**Mixed Precision Training** is **the optimization technique that uses lower-precision floating-point formats (FP16 or BF16) for the majority of training computations while maintaining FP32 precision for critical accumulations — achieving 2-3× training speedup and 50% memory reduction on modern GPUs without sacrificing model accuracy**. **Floating-Point Formats:** - **FP32 (Single Precision)**: 1 sign + 8 exponent + 23 mantissa bits — dynamic range ±3.4×10^38, precision ~7 decimal digits; baseline format for neural network training - **FP16 (Half Precision)**: 1 sign + 5 exponent + 10 mantissa bits — dynamic range ±65,504, precision ~3.3 decimal digits; 2× memory savings and 2× tensor core throughput over FP32 - **BF16 (Brain Float)**: 1 sign + 8 exponent + 7 mantissa bits — same dynamic range as FP32 (±3.4×10^38) but lower precision (~2.4 decimal digits); designed specifically for deep learning to avoid overflow/underflow issues - **TF32 (Tensor Float)**: 1 sign + 8 exponent + 10 mantissa bits — NVIDIA Ampere's automatic FP32 replacement on tensor cores; provides FP32 range with FP16 throughput without code changes **Automatic Mixed Precision (AMP):** - **FP16/BF16 Operations**: matrix multiplications, convolutions, and linear layers run in reduced precision — these operations are compute-bound and benefit most from tensor core acceleration - **FP32 Operations**: reductions (softmax, layer norm, loss computation), small element-wise operations kept in FP32 — these operations are sensitive to precision and contribute negligible compute cost - **Weight Master Copy**: model weights maintained in FP32 and cast to FP16/BF16 for forward/backward — gradient updates applied to FP32 master copy ensuring small updates aren't rounded to zero; 1.5× total memory (FP32 master + FP16 working copy) - **Implementation**: PyTorch torch.cuda.amp.autocast() context manager automatically selects precision per operation — GradScaler handles loss scaling; single-line integration in training loops **Loss Scaling:** - **Gradient Underflow Problem**: FP16 gradients below 2^-24 (~6×10^-8) underflow to zero — many gradient values in deep networks fall in this range, causing training instability or divergence - **Static Loss Scaling**: multiply loss by a constant factor (e.g., 1024) before backward pass, divide gradients by same factor after — shifts gradient values into FP16 representable range; requires manual tuning - **Dynamic Loss Scaling**: start with large scale factor, reduce when inf/nan gradients detected, gradually increase when no overflow — automatically finds optimal scaling; PyTorch GradScaler implements this strategy - **BF16 Advantage**: BF16's full FP32 exponent range eliminates the need for loss scaling entirely — gradients that are representable in FP32 are representable in BF16; simplifies mixed precision training setup **Mixed precision training is the most accessible performance optimization in modern deep learning — requiring minimal code changes while delivering 2-3× speedup and enabling training of larger models within the same GPU memory budget, making it a standard practice for all production training workloads.**