flops (floating point operations)

FLOPs (Floating Point Operations) measure computational cost for training or running neural networks. **Definition**: Count of floating point operations (addition, multiplication, etc.) performed. **Training FLOPs**: Approximately 6ND for transformer training, where N is parameters and D is tokens. Forward and backward pass. **Inference FLOPs**: Approximately 2N per token generated (forward pass only). **PetaFLOP-days**: Common unit for large training runs. GPT-3 trained with approximately 3640 petaflop-days. **GPU specs**: A100: 312 TFLOPS (FP16). H100: 1,979 TFLOPS (FP8). Theoretical vs achieved utilization differs. **MFU (Model FLOP Utilization)**: Ratio of achieved to theoretical FLOPs. Good training achieves 40-60% MFU. **Cost estimation**: Convert FLOPs to GPU-hours, estimate costs. Helps plan training budgets. **Comparison across models**: Normalize by FLOPs to compare efficiency. Model A vs B at same compute. **Precision matters**: Lower precision (FP16, FP8) allows more FLOPs per second but may affect quality. **Industry use**: Standard metric for comparing computational requirements across papers and models.