Mutual Information
Mutual information measures how much knowing one random variable reduces uncertainty about another. Formally it is the KL divergence between the joint distribution of two variables and the product of their separate distributions, which can be written as the entropy of X minus the entropy of X given Y, and it is symmetric: what Y tells about X, X tells about Y equally. It is measured in bits or nats, is never negative, and is exactly zero if and only if the variables are statistically independent. Unlike correlation, which detects only straight-line relationships, mutual information registers any dependence at all, linear or not, so it is a general-purpose score of how much two quantities share.
The classic demonstration of its power is a perfect dependence that correlation misses entirely. Let X be equally likely to be −1, 0 or 1 and let Y be X squared. The two are uncorrelated, since the product of X and Y averages to zero, yet Y is completely determined by X, and the mutual information equals the entropy of Y, about 0.92 bits. A correlation-based screening step would discard X as useless for predicting Y; a mutual-information screen would keep it. Related quantities fill out the picture: the conditional versions, the data processing inequality (processing a variable cannot create information about another), and information gain, which is the mutual information between a feature and the class label used by decision tree learners such as ID3 and C4.5.
Feature selection is the most direct everyday use of mutual information. Ranking features by their estimated mutual information with the target is a filter method: cheap, model-independent, and capable of catching nonlinear relationships, which is why scikit-learn ships mutual_info_classif and mutual_info_regression. For continuous variables these rely on k-nearest-neighbor estimators in the style of Kraskov, Stögbauer and Grassberger's 2004 paper. A known weakness is that ranking each feature alone ignores redundancy, since ten near-copies of one strong feature all score highly. Hanchuan Peng, Fuhui Long and Chris Ding's 2005 minimum-redundancy maximum-relevance method addresses this by balancing a feature's information about the label against its overlap with features already selected.
In representation learning, mutual information is both a motivation and a training signal through the InfoNCE loss. Aaron van den Oord, Yazhe Li and Oriol Vinyals introduced InfoNCE in their 2018 paper on contrastive predictive coding: a model must pick the true positive pair out of a set of N candidates, which include N minus 1 negatives, and minimizing this cross-entropy-style loss maximizes a lower bound on the mutual information between the two views, namely log N minus the loss. The same recipe drives SimCLR (Chen and colleagues, 2020) for images and CLIP (Radford and colleagues, 2021), which aligns images with captions using a symmetric contrastive loss. Tishby, Pereira and Bialek's 1999 information bottleneck frames learning as keeping information about the target while compressing the input.
Mutual information is hard to estimate and easy to over-interpret. In high dimensions with limited data, estimates are biased and demand rapidly growing sample sizes, and the InfoNCE bound cannot exceed log N, so a larger batch can be needed simply to measure a large dependence. Tschannen and colleagues' 2019 study, "On Mutual Information Maximization for Representation Learning," showed that the success of these methods is not explained by the mutual information value itself: encoders with identical bounds can give very different downstream performance, depending on architecture and how views are chosen. A high value also says nothing about causation, and it is not additive across features, so the top-ranked individual features are not necessarily the best set.
Read mutual information as the shared information between two variables, the reduction in uncertainty about one gained from seeing the other: a nonlinear, symmetric alternative to correlation that powers feature screening and, through InfoNCE, much of modern contrastive learning, provided its estimates are treated with caution.