Information Theory

Information theory is the mathematical study of quantifying, storing and communicating information, founded by Claude Shannon's 1948 paper "A Mathematical Theory of Communication" in the Bell System Technical Journal. Shannon defined information in terms of uncertainty removed: a message is informative to the extent that it was unlikely, and the average information of a source is its entropy, measured in bits when logarithms are taken to base 2. His central insight was that the semantic content of a message is irrelevant to the engineering problem of transmitting it; what matters is the statistical structure of the source and the noise in the channel. From that came exact limits on how far data can be compressed and how fast it can be reliably sent, limits that have guided the design of every modern communication and storage system.

Shannon's two foundational theorems give the field its architecture: the source coding theorem fixes the best possible compression, and the noisy-channel coding theorem fixes the best possible reliable transmission rate. The first says that a source with entropy H bits per symbol cannot be losslessly compressed to fewer than H bits per symbol on average, but can be compressed arbitrarily close to H; Huffman coding (1952) gives an optimal symbol-by-symbol code, and arithmetic coding approaches the limit more closely. The second says that every noisy channel has a capacity C, and that information can be sent at any rate below C with error probability made arbitrarily small by clever coding, but at no rate above C. For a band-limited channel with Gaussian noise, the Shannon–Hartley formula gives capacity as bandwidth × log2(1 + signal-to-noise ratio). The 1948 proof was non-constructive, and it took decades to find practical codes close to the limit: Gallager's low-density parity-check codes of the early 1960s, rediscovered later, and the turbo codes of Berrou, Glavieux and Thitimajshima (1993), now underlie wireless standards and satellite links.

A small set of quantities defined on probability distributions carries the whole theory: entropy, joint and conditional entropy, mutual information, cross-entropy and the Kullback–Leibler divergence. Mutual information I(X; Y) measures how much knowing one variable reduces uncertainty about another, in bits, and equals zero exactly when they are independent; it generalizes correlation by capturing any dependence, linear or not. The KL divergence (Kullback and Leibler, 1951) measures the extra bits needed when a code is designed for a distribution Q but data really follow P, and it is not symmetric, so it is not a distance. Cross-entropy equals entropy plus KL divergence, which is why minimizing cross-entropy loss pulls a model's distribution toward the data, and the same quantity is the negative log-likelihood used in maximum-likelihood-estimation. The detailed definition of entropy is treated in its own entry; here it matters as the base from which the other measures are built.

Information theory now reaches well beyond telecommunications, and in machine learning its concepts are everywhere. Cross-entropy is the standard classification loss and the training objective of language models; perplexity, the exponentiated cross-entropy per token, is the usual language-model metric; KL divergence appears in variational autoencoders, in expectation-maximization's lower bound and in the policy constraints of reinforcement learning from human feedback; mutual information drives feature selection, contrastive objectives such as InfoNCE and Tishby, Pereira and Bialek's 1999 information bottleneck. Jorma Rissanen's minimum description length principle (1978) turns model selection into compression. Outside machine learning, the field underpins error-correcting codes in storage, DNA sequencing and cryptography, where Shannon's 1949 paper on secrecy systems proved that a one-time pad provides perfect secrecy, and the standard textbook is Cover and Thomas's "Elements of Information Theory."

Misunderstandings usually stem from the word "information" itself. In Shannon's sense information measures unpredictability and not meaning, truth or usefulness: a random string has maximal entropy and no significance, and a true, important message that the receiver already expected carries few bits. Entropy is relative to a model of the source, so the same text has different entropy under a unigram model and under a language model, and estimating it from finite data is biased. Capacity theorems are asymptotic: they promise reliable transmission with long blocks and unbounded complexity, so finite-latency systems pay a real penalty. Finally, applying the theory to brains, ecosystems or economics requires first defining a source, a channel and a probability model, and the numbers are only as meaningful as those modeling choices; there is more than one way to measure "information," and Shannon's is a single, though foundational, one.

Read information theory as the science of surprise and its price: every message has a measurable minimum cost in bits, every channel a measurable maximum rate, and almost every modern system that compresses, transmits, learns or predicts is operating somewhere between those two Shannon limits.