Information Theory Shannon
"A Mathematical Theory of Communication," by Claude E. Shannon, appeared in the Bell System Technical Journal in two installments, July and October 1948 (volume 27, pages 379–423 and 623–656), and is widely regarded as the founding document of information theory. The paper is treated here as an event: a specific publication that took scattered engineering ideas about telegraphy, telephony and cryptography and recast them as a single mathematical theory with theorems, definitions and limits. The general field that grew from it has its own entry, and Shannon's life is covered in his biography.
The paper's governing move was to define the engineering problem of communication precisely and to set aside meaning. Shannon pictured a source producing messages, a transmitter encoding them into signals, a noisy channel, and a receiver reconstructing the message, and he declared that the semantic aspects of messages are irrelevant to this engineering problem: what matters is that the actual message is one selected from a set of possible messages. That choice let him measure information by how surprising a message is under a probability model, giving the entropy of a source, H, as the negative sum over messages of each probability times the logarithm of that probability, in base two to count in binary digits. The paper also made the word "bit" famous in print, crediting John Tukey with the contraction of "binary digit." Shannon built on earlier work by Harry Nyquist in the 1920s on telegraph signaling speed and by Ralph Hartley in 1928 on measuring information by the number of possible selections, but neither had treated probability and noise as central.
The first major theorem concerns source coding: the entropy of a source is the minimum average number of bits per symbol needed to represent it without loss. A source whose symbols are predictable can be compressed, and one whose symbols are uniformly random cannot. English text, with its uneven letter frequencies and strong dependencies between letters and words, has entropy well below the cost of a naive fixed-length code, which is the principle behind every lossless compressor from Huffman coding, developed by David Huffman in 1952, to modern arithmetic coding. Shannon himself estimated the redundancy of English in his 1951 paper "Prediction and Entropy of Printed English," in which human guessers predicted the next letter of a text, an experiment that anticipates language modeling as a prediction task.
The second and more startling result was the noisy-channel coding theorem. Engineers had assumed that reducing errors on a noisy line required slowing down or boosting power so that the error rate fell only as transmission rate fell. Shannon proved that every channel has a capacity, C, and that for any rate below C there exist codes making the probability of error as small as desired, while above C reliable communication is impossible. For a channel with additive Gaussian noise the capacity is C = B log2(1 + S/N), where B is bandwidth and S/N the signal-to-noise ratio, a formula known as the Shannon–Hartley theorem. The proof was one of existence, using random coding arguments, and gave no practical recipe, so it set a target that took decades to approach: Hamming codes and Reed–Solomon codes in the meantime, then turbo codes in 1993 by Claude Berrou, Alain Glavieux and Punya Thitimajshima, and low-density parity-check codes, rediscovered in the 1990s after Robert Gallager's 1960s thesis, which brought systems within a fraction of a decibel of the limit. A further structural insight was the separation of source coding from channel coding, which lets compression and error correction be designed independently.
The paper reached beyond engineering within a year, and a few misunderstandings followed it. Warren Weaver's essay and Shannon's paper were published together as a book by the University of Illinois Press in 1949, and the framework spread into linguistics, psychology, biology and cybernetics. Shannon himself cautioned in 1956 that the theory had become something of a bandwagon and was not a universal key. The most common error is to read information in this sense as meaning or truth: a random string carries more Shannon information than a meaningful sentence of the same length. A second is to treat Shannon entropy and thermodynamic entropy as simply identical; they share a mathematical form and are deeply connected in statistical mechanics, but they are not interchangeable quantities in general. Its modern descendants are everywhere in machine learning: cross-entropy is the standard training loss for classifiers and language models, a model's perplexity is an exponentiated entropy, and compression is routinely used as a lens on how well a model has captured the structure in data.
Read the 1948 paper as the moment communication acquired a hard physical limit and a way to meet it: by defining information as the reduction of uncertainty rather than as meaning, Shannon turned noise from an unavoidable nuisance into a quantity with a precise ceiling that engineering could chase and that learning systems still use as a yardstick.