Maximum Likelihood Estimation

Maximum likelihood estimation (MLE) is a method for fitting a probabilistic model by choosing the parameter values that make the observed data most probable under that model: the likelihood function, which is the joint probability (or density) of the data viewed as a function of the parameters, is maximized, and the maximizer is the estimate. Ronald Fisher introduced and named the method in a series of papers between 1912 and 1922, most fully in "On the Mathematical Foundations of Theoretical Statistics" (1922), building on earlier work by Gauss and Laplace. It is the default way parameters get estimated across classical statistics and, under the name "minimize the negative log-likelihood," across most of machine learning.

Maximizing the log-likelihood instead of the likelihood is a harmless trick that does most of the practical work, because it turns a product over independent observations into a sum that can be differentiated, summed in mini-batches and computed without numerical underflow. For n independent observations the log-likelihood is a sum of n terms, and setting its gradient (the score) to zero gives the estimating equations. A tiny worked case: seven heads in ten coin flips give a Bernoulli log-likelihood of 7 × log(p) + 3 × log(1 − p), whose maximum is p = 7/10, the sample proportion. For a normal distribution the MLE of the mean is the sample mean, and the MLE of the variance divides by n rather than n − 1, which makes it slightly biased low in small samples, a reminder that maximum likelihood optimizes fit and not unbiasedness. For logistic regression, neural networks and most real models no closed form exists, so the maximum is found numerically by Newton-type iteration (iteratively reweighted least squares in generalized linear models) or by stochastic gradient descent.

The method's appeal rests on large-sample guarantees: under regularity conditions the MLE is consistent, asymptotically normal, and asymptotically efficient, reaching the Cramér–Rao lower bound on variance. The curvature of the log-likelihood at its peak, measured by the Fisher information, tells how sharply the data pin the parameter down, and its inverse approximates the estimator's variance, which is how standard errors and Wald confidence intervals are produced. Likelihood-ratio tests, the Akaike information criterion (Akaike, 1974) and the Bayesian information criterion (Schwarz, 1978) are all built from maximized log-likelihoods. The invariance property is also convenient: if the MLE of a parameter is known, the MLE of any function of it is that function of the estimate.

Many familiar loss functions are maximum likelihood in disguise, which is why the principle unifies so much of machine learning. Minimizing mean squared error corresponds to maximizing the likelihood of Gaussian noise with constant variance; minimizing cross-entropy for classification, including the next-token loss used to train language models, is exactly minimizing the negative log-likelihood of the labels under the model's predicted probabilities, an identity that connects to the cross-entropy of information-theory. Where the likelihood includes unobserved variables, a direct maximization is awkward, and the expectation-maximization algorithm and gaussian-mixture-model fitting exist to handle that case. The Bayesian counterpart, covered under bayesian-inference, multiplies the likelihood by a prior and keeps the whole distribution, and the maximum a posteriori estimate is MLE with a penalty term.

The standard misconceptions are that the likelihood is the probability of the parameter and that the maximum is always trustworthy. The likelihood function is not a probability distribution over parameters: it does not integrate to one, it changes under reparameterization differently from a density, and the probability of the parameter given the data requires a prior and bayes-theorem. In practice MLE overfits when parameters are many and data few, since it chases noise; a mixture likelihood can be unbounded when a component collapses onto a single point, so the "maximum" is infinite and meaningless; a likelihood can have several local maxima; and in logistic regression with perfectly separable classes the maximum is at infinity. Remedies include regularization, early stopping, multiple random starts, and checking that the fitted model could plausibly have generated the data, because maximum likelihood finds the best model within the chosen family, not a good model if the family is wrong.

Read maximum likelihood estimation as asking which version of the model would have been least surprised by the data, a principle so general that most training losses are its special cases, and so greedy that it needs regularization, diagnostics or a prior whenever data are scarce.