FP32 (Single-Precision Floating Point) is the 32-bit numerical format that serves as the baseline precision for neural network training — using 1 sign bit, 8 exponent bits, and 23 mantissa bits to represent numbers with ~7 decimal digits of precision across a range of ±3.4×10³⁸, providing the numerical stability needed for gradient computation and weight updates while consuming 4 bytes per parameter, making a 7B parameter model require 28 GB of memory in FP32 representation.

What Is FP32?

• Definition: The IEEE 754 single-precision floating-point format — 32 bits total with 1 sign bit (positive/negative), 8 exponent bits (range: 2⁻¹²⁶ to 2¹²⁷), and 23 mantissa bits (~7 decimal digits of precision). The standard numerical format for scientific computing and the default training precision for neural networks.
• Training Baseline: FP32 is the "gold standard" precision for training — all gradients, weights, activations, and optimizer states are computed and stored in FP32 by default, providing sufficient precision for the small gradient updates that drive learning.
• Memory Cost: 4 bytes per value — a 7B parameter model requires 28 GB just for weights in FP32, plus 2-3× more for optimizer states (Adam stores momentum and variance in FP32), making FP32 training memory-intensive.
• Master Weights: In mixed-precision training, a FP32 copy of all weights is maintained as "master weights" — FP16/BF16 is used for forward/backward computation, but weight updates are applied to the FP32 master copy to prevent precision loss from accumulating small gradient updates.

FP32 vs. Other Precisions

FP32 in the ML Workflow

• Training: FP32 master weights + FP16/BF16 compute = mixed-precision training — 2× speedup on tensor cores with minimal accuracy loss. FP32 accumulation prevents precision loss in reductions.
• Inference: FP32 is rarely used for inference in production — models are quantized to FP16, INT8, or INT4 for 2-8× memory reduction and faster execution.
• TF32 (Tensor Float 32): NVIDIA A100/H100 tensor cores transparently compute FP32 operations at TF32 precision (8-bit exponent, 10-bit mantissa) — providing 8× speedup over true FP32 with same range but reduced precision, enabled by default.

FP32 is the numerical foundation of neural network training — providing the precision and range needed for stable gradient computation and weight updates, while mixed-precision techniques and inference quantization reduce its memory and compute costs by using lower-precision formats where full FP32 accuracy is not required.