Kl Divergence
KL divergence, short for Kullback-Leibler divergence and also called relative entropy, measures how much one probability distribution differs from another as the expected extra surprise incurred when outcomes truly drawn from p are described using a model q. In symbols it is the sum over outcomes of p times the logarithm of (p divided by q), and it equals the cross-entropy of p and q minus the entropy of p. Solomon Kullback and Richard Leibler introduced it in their 1951 paper "On Information and Sufficiency." It is never negative (a result known as Gibbs' inequality), it is zero only when the two distributions are identical, and it is infinite if q gives zero probability to something p considers possible.
KL divergence is asymmetric, and the direction chosen decides what behavior it rewards. Take p = (0.5, 0.5) and q = (0.9, 0.1): the divergence from p to q is about 0.51 nats, while the divergence from q to p is about 0.37 nats. Because of this it is not a distance in the mathematical sense and does not satisfy the triangle inequality, so it is called a divergence. Minimizing the forward direction, with the data distribution as the first argument, forces the model to cover every region where the data has mass, producing broad, mass-covering fits that may spill into empty regions. Minimizing the reverse direction lets the model ignore some modes of the data as long as what it does produce is plausible, producing sharp, mode-seeking fits. Symmetric alternatives such as the Jensen-Shannon divergence, which averages the divergences to a midpoint distribution, exist and underlie the original GAN objective.
In variational autoencoders KL divergence is the regularizer that keeps the learned latent space well-behaved. Diederik Kingma and Max Welling's 2013 paper "Auto-Encoding Variational Bayes" trains an encoder that outputs a distribution over latent codes and a decoder that reconstructs the input; the training objective, the evidence lower bound, is the reconstruction term minus the KL divergence from the encoder's distribution to a fixed prior, usually a standard normal. For Gaussians this KL term has a closed form in the encoder's means and variances, so it can be computed and differentiated exactly. Pushing it down pulls codes toward the prior so that sampling from the prior later produces sensible outputs, while a weight on it, as in the beta-VAE variant, trades reconstruction quality against disentangled, smooth latent structure. A failure mode called posterior collapse occurs when the KL term wins and the decoder ignores the latent code.
In reinforcement learning from human feedback, a KL penalty tethers the tuned model to its starting point. When a language model is optimized against a reward model, as in Ziegler and colleagues' 2019 work on fine-tuning from human preferences, Stiennon and colleagues' 2020 summarization work, and OpenAI's 2022 InstructGPT, the reward is reduced by a coefficient times the KL divergence between the new policy and a frozen reference model, usually estimated token by token on sampled text. Without it the policy can drift into gibberish that exploits flaws in the reward model, a failure called reward hacking, so the penalty keeps outputs fluent and close to the supervised model. Direct preference optimization, a later method, builds the same implicit KL constraint into its loss, and the trust-region idea goes back to Schulman and colleagues' 2015 TRPO, which bounds the KL between successive policies.
KL divergence also appears in distillation, drift monitoring and model comparison, with a few recurring traps. Geoffrey Hinton and colleagues' 2015 knowledge distillation trains a small student to match a large teacher's softened output distribution, which is a KL minimization. Data teams use it to quantify drift between a training distribution and live traffic. The traps are practical: it is unbounded and explodes if the second distribution assigns near-zero probability to events the first considers possible, so estimates need smoothing; histogram-based estimates depend heavily on binning; and its units (nats or bits) must be reported. Treating it as a symmetric distance, or reading a small value as proof two models behave alike on rare but important inputs, are common errors.
Read KL divergence as the cost in extra bits of believing q when the truth is p: a one-directional, non-negative gap that is zero only for a perfect match, whose choice of direction shapes what a model learns and which holds latent spaces and fine-tuned language models near a prior or reference.