Cross-Entropy
Cross-entropy measures the average number of bits (or nats) needed to encode events that really follow a distribution p when the code has been designed for a different distribution q. It equals the sum over outcomes of minus p times the logarithm of q, and it is always at least as large as the entropy of p itself, with equality only when q matches p exactly. That gap, the extra cost of believing the wrong model, is the KL divergence from p to q, so cross-entropy equals entropy plus KL divergence. In machine learning, p is the truth (the observed label or next word) and q is the model's predicted probability, which makes cross-entropy the natural score for how badly a model's probabilities fit reality.
For classification the formula collapses to something simple: with a one-hot label, cross-entropy is just minus the logarithm of the probability the model assigned to the correct class. If a model gives the right class a probability of 0.25, the loss is the natural log of 4, about 1.39 nats or exactly 2 bits; if it gives 0.9 the loss is about 0.105 nats; if it gives 0.001 the loss is about 6.9. The penalty grows without bound as confidence in the wrong answer grows, which is deliberate: a confidently wrong prediction is punished far more than a mildly wrong one. Binary cross-entropy, also called log loss, is the two-class version, summing a term for each side, and the same idea extends to multi-label problems by applying it per label.
Minimizing cross-entropy is the same thing as maximum likelihood estimation, which is why it became the default classification loss. Maximizing the probability a model assigns to the training labels is equivalent to minimizing the average negative log-probability, and that average is exactly the cross-entropy between the empirical label distribution and the model. When cross-entropy is paired with a softmax output, the gradient of the loss with respect to the pre-softmax scores is simply the predicted probability minus the target, a clean, well-scaled signal that does not vanish when the model is confidently wrong. This is a central practical advantage over squared error on probabilities, which gives small gradients exactly where a saturated sigmoid or softmax is most mistaken.
In language models, cross-entropy is the training objective and perplexity is its exponentiated form. A causal language model such as the GPT family is trained to minimize the average negative log-probability of each next token given the preceding ones, and perplexity is the exponential of that average (base e for nats, base 2 for bits). It can be read as the effective number of equally likely choices the model is torn between at each step: a perplexity of 20 means the model is, on average, as uncertain as if it were choosing uniformly among 20 tokens. Because perplexity is computed per token, values are not comparable across different tokenizers or vocabularies, and bits per byte or bits per character are used when models need a tokenizer-independent comparison.
Several pitfalls are worth knowing. Implementations compute the loss from logits using the log-sum-exp trick (as in PyTorch's CrossEntropyLoss, which combines log-softmax with negative log-likelihood) rather than taking the log of a computed softmax, since a probability that underflows to zero would produce an infinite loss. Mixing up units causes confusion: libraries report nats, papers sometimes report bits. Label smoothing, used in Szegedy and colleagues' 2016 Inception-v3 work and in many transformers, replaces one-hot targets with slightly softened ones so the model is not pushed toward infinite confidence. Finally, low cross-entropy measures probability quality, not accuracy: a model can have good accuracy and poorly calibrated probabilities, and the loss is still sensitive to that miscalibration.
Read cross-entropy as the price in bits of describing reality with the model's probabilities instead of the true ones: for a classifier it is the negative log of the probability given to the right answer, for a language model its exponential is perplexity, and driving it down is exactly maximum-likelihood learning.