Mixed Precision Training is the optimization technique that uses lower-precision floating-point formats (FP16 or BF16) for the majority of training computations while maintaining FP32 precision for critical accumulations — achieving 2-3× training speedup and 50% memory reduction on modern GPUs without sacrificing model accuracy.

Floating-Point Formats:

• FP32 (Single Precision): 1 sign + 8 exponent + 23 mantissa bits — dynamic range ±3.4×10^38, precision ~7 decimal digits; baseline format for neural network training
• FP16 (Half Precision): 1 sign + 5 exponent + 10 mantissa bits — dynamic range ±65,504, precision ~3.3 decimal digits; 2× memory savings and 2× tensor core throughput over FP32
• BF16 (Brain Float): 1 sign + 8 exponent + 7 mantissa bits — same dynamic range as FP32 (±3.4×10^38) but lower precision (~2.4 decimal digits); designed specifically for deep learning to avoid overflow/underflow issues
• TF32 (Tensor Float): 1 sign + 8 exponent + 10 mantissa bits — NVIDIA Ampere's automatic FP32 replacement on tensor cores; provides FP32 range with FP16 throughput without code changes

Automatic Mixed Precision (AMP):

• FP16/BF16 Operations: matrix multiplications, convolutions, and linear layers run in reduced precision — these operations are compute-bound and benefit most from tensor core acceleration
• FP32 Operations: reductions (softmax, layer norm, loss computation), small element-wise operations kept in FP32 — these operations are sensitive to precision and contribute negligible compute cost
• Weight Master Copy: model weights maintained in FP32 and cast to FP16/BF16 for forward/backward — gradient updates applied to FP32 master copy ensuring small updates aren't rounded to zero; 1.5× total memory (FP32 master + FP16 working copy)
• Implementation: PyTorch torch.cuda.amp.autocast() context manager automatically selects precision per operation — GradScaler handles loss scaling; single-line integration in training loops

Loss Scaling:

• Gradient Underflow Problem: FP16 gradients below 2^-24 (~6×10^-8) underflow to zero — many gradient values in deep networks fall in this range, causing training instability or divergence
• Static Loss Scaling: multiply loss by a constant factor (e.g., 1024) before backward pass, divide gradients by same factor after — shifts gradient values into FP16 representable range; requires manual tuning
• Dynamic Loss Scaling: start with large scale factor, reduce when inf/nan gradients detected, gradually increase when no overflow — automatically finds optimal scaling; PyTorch GradScaler implements this strategy
• BF16 Advantage: BF16's full FP32 exponent range eliminates the need for loss scaling entirely — gradients that are representable in FP32 are representable in BF16; simplifies mixed precision training setup

Mixed precision training is the most accessible performance optimization in modern deep learning — requiring minimal code changes while delivering 2-3× speedup and enabling training of larger models within the same GPU memory budget, making it a standard practice for all production training workloads.