nan

NaN (Not a Number) and Inf (Infinity) values appearing during training indicate numerical instability that must be diagnosed and resolved to enable successful model training. Common causes: division by zero (normalizing with zero variance, empty batches), log of zero or negative (log-probabilities, cross-entropy edge cases), overflow (exponentials growing unbounded, large gradients), underflow-to-zero (very small values truncated, then divided by), exploding gradients (values exceeding float range), and ill-conditioned matrices (inverting near-singular matrices). Diagnosis: add checks for NaN/Inf after each operation, use torch.autograd.detect_anomaly() or TensorFlow debugging, and trace which layer/operation first produces NaN. Fixes: lower learning rate (reduce gradient magnitude), gradient clipping (cap gradient norm), add epsilon to denominators (1e-8 stability), use log-sum-exp (numerical stability for log-softmax), and verify data (NaN in inputs propagates). Scaling strategies: mixed precision with loss scaling, proper normalization (LayerNorm, BatchNorm), and careful initialization (avoiding extreme values). Persistent NaN often indicates code bugs (incorrect reshape, wrong dimension), while intermittent NaN suggests edge cases in data or numerical boundary conditions. Proper numerical hygiene prevents training instabilities.