Entropy

Entropy, in information theory, is the average amount of surprise in the outcomes of a random variable, measured in bits when logarithms are taken base 2. For a distribution with probabilities p1, p2, and so on, it equals the sum over outcomes of p times log2 of (1/p), which can also be read as the expected number of yes/no questions needed to pin down an outcome when questions are asked in the best possible order. Claude Shannon defined it in his 1948 paper "A Mathematical Theory of Communication," and the quantity is a property of a probability distribution, not of any single message: a fair coin has exactly 1 bit of entropy, a fair eight-sided die has 3 bits, and a coin that always lands heads has 0 bits because nothing it does is surprising.

Entropy is the hard floor on lossless compression, which is what turns an abstract formula into an engineering limit. Shannon's source coding theorem says that no lossless code can use fewer bits per symbol, on average, than the entropy of the source, and that codes exist which come arbitrarily close. A biased coin that lands heads 90% of the time has entropy of about 0.47 bits per flip, so a long run of such flips can be squeezed to under half its naive one-bit-per-flip size, and the less uniform a source is, the more it can be compressed. David Huffman's 1952 prefix codes and later arithmetic coding approach this limit in practice, and the same logic explains why random-looking data, whose entropy is already near its maximum, does not compress at all. Shannon's 1951 experiments with human guessers estimated the entropy of printed English at roughly one bit per letter, far below the several bits a letter would need if all characters were equally likely.

Each outcome carries surprise equal to the logarithm of the inverse of its probability, and entropy is the average of that surprise. Rare events carry many bits and common events few, and the logarithm is what makes information from independent events add up: two independent fair coin flips carry 1 + 1 = 2 bits, matching a single fair four-sided die. Entropy is largest when all outcomes are equally likely, reaching log2 of the number of outcomes, and falls to zero as the distribution concentrates on one outcome. Using natural logarithms instead changes the unit from bits to nats, a pure rescaling by the constant ln 2, which is why machine learning code, whose libraries default to natural logs, reports losses in nats while textbooks often quote bits.

Thermodynamic entropy shares a name and a formal resemblance but answers a different question, and conflating them is the most common misconception. Boltzmann and Gibbs defined entropy in physics as a measure of the number of microscopic arrangements consistent with a macroscopic state, scaled by the Boltzmann constant and measured in joules per kelvin. The mathematical form of Gibbs entropy matches Shannon's, and the two connect deeply in statistical mechanics, but information entropy is a dimensionless property of a probability model and says nothing about heat, and neither is simply "disorder." A closely related error is treating entropy as a property of one string: the string 0101010101 can come from a high-entropy source by chance, and the per-string notion is Kolmogorov complexity, a separate idea that is uncomputable in general.

Entropy appears throughout machine learning as a measure of uncertainty, an objective, and a regularizer. Decision tree learners such as Ross Quinlan's ID3 from the 1980s choose splits that most reduce the entropy of the class labels, a quantity called information gain. A classifier's predicted distribution has high entropy when it is unsure and low entropy when it is confident, so predictive entropy is a standard crude uncertainty score. Reinforcement learning methods such as soft actor-critic add an entropy bonus to keep policies exploratory, and language models are judged by how few bits per token or per byte they need, which is the cross-entropy against the real text distribution and can never fall below the true entropy of the text. The related quantities cross-entropy, KL divergence and mutual information are all built from this same logarithmic surprise.

Several practical pitfalls follow from entropy being defined relative to a model. Estimating entropy from a finite sample is biased downward when many outcomes are rarely observed, so naive plug-in estimates of the entropy of a large vocabulary or a continuous variable can mislead. Differential entropy, the continuous analogue obtained by integrating over a density, can be negative and changes under rescaling, so it lacks the clean interpretation of the discrete version. And the "entropy" of English quoted from Shannon's experiments depends on the context a model or human is allowed to see: more context means lower measured entropy, which is exactly the gap modern language models try to close.

Read entropy as the average surprise of a source, the number of bits per outcome that no lossless code can beat: a measure of how unpredictable a distribution is, not of physical disorder, and the logarithmic yardstick from which cross-entropy, KL divergence and mutual information are all built.