Exploding Gradient
An exploding gradient is a training failure in which the gradients of the loss grow exponentially large as they are propagated backward through many layers or time steps, producing enormous parameter updates that destabilize or destroy learning. The usual symptoms are loss values that spike, weights that become huge, and eventually infinities or NaN values that poison the whole model. It is the mirror image of the vanishing gradient problem, where the same products shrink toward zero, and both arise from the same arithmetic of repeated multiplication.
The cause is the chain rule applied across a long sequence. In a recurrent neural network the same weight matrix is applied at every time step, so the gradient of the loss with respect to an early hidden state contains a product of T Jacobian matrices, one per step. If the largest singular value of those factors stays above 1, the product grows roughly like that value raised to the power T; if it stays below 1, it decays at the same exponential rate. Bengio, Simard and Frasconi analyzed this tension for learning long-term dependencies in 1994, and Hochreiter had described the vanishing side in his 1991 thesis. Pascanu, Mikolov and Bengio sharpened the picture in "On the difficulty of training recurrent neural networks" (ICML 2013), giving a geometric view in which the loss surface contains steep cliff-like walls that a single gradient step can overshoot catastrophically.
Gradient clipping is the standard and cheap remedy. Pascanu and colleagues proposed norm clipping: if the norm of the gradient exceeds a chosen threshold, rescale the whole gradient so its norm equals the threshold, which preserves the direction while limiting the step length. A simpler variant clips each component separately, but that changes the direction. Large transformer training commonly clips the global gradient norm at around 1.0, and clipping events logged during a run are a useful diagnostic, since a sudden rise in how often it triggers often precedes a loss spike. Other defenses include careful initialization such as orthogonal or Xavier-style schemes, lower learning rates, warmup, normalization layers, and residual connections, which give gradients a path whose Jacobian is close to the identity.
Exploding and vanishing gradients are different problems that need different fixes, and one tool does not cure both. Clipping addresses only the explosion; it does nothing when a gradient has shrunk to nothing, because there is no signal to rescale. Gated architectures such as the LSTM, introduced by Hochreiter and Schmidhuber in 1997, were designed mainly to combat vanishing by letting information pass along an additive path, yet LSTMs can still explode and are routinely trained with clipping. Truncated backpropagation through time limits how many steps a gradient crosses, which bounds both effects at the cost of ignoring very long dependencies.
The phenomenon is not confined to old recurrent networks. Very deep feed-forward networks without residual connections suffer it, and large language model runs are known to show sudden loss spikes from numerical instability, mixed-precision overflow or bad batches. A common misconception is that a NaN always means a bug in the code; it can equally be a legitimate explosion, so checking gradient norms, not just the loss, is the first diagnostic step. Another pitfall is setting the clipping threshold so low that it is always active, which silently turns the optimizer into normalized gradient descent and throws away useful magnitude information.
Read exploding gradient as the runaway side of repeated multiplication: when many chained factors each exceed one, the learning signal swells exponentially until a single step flings the model off the loss surface, and clipping works by capping the step while leaving the direction intact.