Bayesian Inference

Bayesian inference is the workflow of learning about unknown quantities by starting from a probability model (a prior over the unknowns and a likelihood describing how data would arise from them), conditioning on the data actually observed, and reporting the result as a probability distribution rather than a single best guess. Where bayes-theorem is the one-line identity that relates the pieces, Bayesian inference is the whole procedure built around it: specify, condition, check, predict, and then repeat when more data arrive. Its roots run from Thomas Bayes's essay, published posthumously in 1763 through Richard Price, to Pierre-Simon Laplace, who independently developed the idea and applied it to problems ranging from the sex ratio of births to the masses of planets, and the approach was known for most of the nineteenth century as "inverse probability."

The update loop is the heart of the method: choose a model, state a prior, condition on data to obtain a posterior, check that the fitted model can reproduce the data, and then use the posterior to make predictions or decisions. The details of the first two steps belong to prior-probability and the interpretation of the output belongs to posterior-probability; what is distinctive about inference as a whole is the ordering and the recursion. Because conditioning is associative, feeding data in one batch or one observation at a time yields the same posterior, and yesterday's posterior simply becomes today's prior. That sequential property is why recursive Bayesian filtering (the Kalman filter is the Gaussian special case) tracks spacecraft and phones, why Thompson sampling uses a running posterior to decide which slot machine or web page variant to try next, and why online A/B testing platforms can report the probability that variant B beats variant A at any moment.

The prior-times-likelihood recipe is simple to write down but rarely simple to compute, and the difficulty of the normalizing constant (the marginal likelihood, an integral over every possible parameter value) is what shaped the field's history. Only a few conjugate pairs, like the Beta-Binomial update in the diagram or the Normal-Normal pair, give closed-form posteriors. Everything else needs approximation: Laplace approximations, variational inference (which turns inference into optimization and underlies variational autoencoders), and above all markov-chain-monte-carlo, whose spread through statistics after Gelfand and Smith's 1990 paper made richly structured hierarchical models practical. Modern probabilistic programming systems such as Stan, PyMC and Turing.jl let an analyst write down the model and leave the sampling to an engine, so the main intellectual work shifts to modeling and diagnostics.

What Bayesian inference offers over a single maximum-likelihood-estimation fit is a full account of uncertainty, propagated automatically into every downstream quantity. Predictions average over the posterior instead of plugging in one parameter value, partial pooling in hierarchical models lets small groups borrow strength from large ones, and Bayes factors or posterior predictive comparisons weigh competing models. In machine learning the same logic appears as Bayesian neural networks, Gaussian processes, probabilistic matrix factorization and calibrated uncertainty for medical or scientific predictions, although exact posteriors over millions of neural network weights are out of reach and practical methods (deep ensembles, Monte Carlo dropout, Laplace approximations) are approximations of uneven quality. With large samples and a well-behaved model, the Bernstein–von Mises theorem says the posterior concentrates around the maximum-likelihood estimate and is approximately normal, so the two schools often agree numerically while differing in what the numbers are said to mean.

The most common misconception is that Bayesian inference is "just subjective guessing" or, in the opposite error, that it is an objective machine that turns data into truth; in fact its conclusions are conditional on the model as a whole, not only the prior. The likelihood is usually a bigger assumption than the prior: a posterior from a badly misspecified model can be tightly concentrated and confidently wrong. That is why the modern Bayesian workflow, as laid out by Gelman and colleagues, insists on prior predictive checks before fitting, convergence diagnostics (R-hat and effective sample size) during fitting, posterior predictive checks after fitting, and sensitivity analysis over reasonable alternative priors. A second pitfall is treating the posterior as a verdict: it is a statement about plausibility within the model, and the decision still needs a loss function, a cost of being wrong in each direction.

Read Bayesian inference as the discipline of keeping a running, honestly uncertain belief that is revised by evidence in exactly the amount the model says the evidence deserves, with the prior, the likelihood and the checking step all open to inspection.