Prior Probability

A prior probability (or prior distribution, when the unknown is a continuous quantity) is the probability assigned to an unknown before the current data are taken into account, expressing what is already believed, known or assumed about it. It is the first input to Bayesian analysis: in the update workflow described under bayesian-inference it is multiplied by the likelihood to produce a posterior, and the interpretation of that output is covered under posterior-probability. The focus here is the prior itself, meaning what kinds of priors exist, how they are chosen, how experts' knowledge is turned into one, and how much they matter. The word "prior" means prior to this data, not necessarily earlier in time or free of evidence; a prior can be built from earlier studies, physical constraints, or a deliberate decision to say as little as possible.

Priors range along a spectrum from flat to strongly informative, and the modern consensus sits in the middle: weakly informative priors that rule out absurd values without pretending to precision. A flat or "noninformative" prior, the principle of indifference that Laplace used, treats all values as equally likely, but flatness is not invariant: a uniform prior on a probability p is not uniform on the odds, and a flat prior on a variance is not flat on a standard deviation. Harold Jeffreys's 1946 invariant prior, proportional to the square root of the Fisher information, addresses this reparameterization problem, and E. T. Jaynes advocated maximum-entropy priors, which choose the least committal distribution consistent with stated constraints. Andrew Gelman's 2006 paper on prior distributions for variance parameters in hierarchical models popularized weakly informative choices such as half-Cauchy or half-normal priors on scale parameters, and the Stan project's prior-choice recommendations follow the same philosophy. A genuinely informative prior, built from earlier trials or physics, is appropriate when data are scarce and knowledge is real.

Prior elicitation is the craft of converting expert judgment into a probability distribution, and it is a discipline with documented methods and documented biases. O'Hagan and colleagues' 2006 book "Uncertain Judgements: Eliciting Experts' Probabilities" is the standard reference, and the Sheffield Elicitation Framework (SHELF) provides a structured protocol in which experts give quantiles or plausible ranges for observable quantities, not for abstract parameters, and a distribution is fitted to their statements and fed back for revision. Experts are well known to be overconfident, anchoring on a first guess and producing intervals that are too narrow, so elicitation methods ask for medians and tail quantiles, use feedback and, when several experts are involved, pool the results. Priors can also be checked against the data-generating process through prior predictive simulation, which turns an opaque choice such as a standard deviation of 10 on a logistic-regression coefficient (implying near-certain outcomes) into an intelligible, falsifiable statement.

The link between priors and regularization is one of the most useful bridges between Bayesian and mainstream machine learning. The maximum a posteriori estimate under a Gaussian prior on weights is exactly L2-penalized (ridge) estimation, and a Laplace prior corresponds to L1 (lasso) penalization, so weight decay in a neural network can be read as a prior belief that weights are small. Hierarchical priors, where the prior's own parameters are learned from related groups, power partial pooling; and conjugate priors, chosen so that prior and posterior belong to the same family (Beta for Binomial, Gamma for Poisson, Normal for Normal), keep arithmetic closed-form. Compared with the likelihood in maximum-likelihood-estimation, the prior is the extra term that allows the analyst to stabilize estimates when the data alone are thin, for example when a category has two observations or a regression has more predictors than rows.

A prior's influence fades as data accumulate, but not uniformly, and a prior that assigns zero probability to something can never be overturned by any amount of evidence. With plentiful data and a regular model, reasonable priors lead to practically the same conclusions (the washing-out result behind the Bernstein–von Mises theorem), but in small samples, in weakly identified parameters, and in the tails of the distribution they can dominate. The standard pitfalls are treating a flat prior as automatically objective, choosing a vague prior such as a Normal with enormous variance that quietly puts almost all its mass on absurd values, picking the prior after looking at the posterior until the result is pleasing, and forgetting that "uninformative" priors can even produce improper posteriors. Good practice is to state priors openly, justify them, and report how much the answer changes when they do.

Read a prior as an explicit, inspectable statement of what was assumed before the data spoke, strongest when it encodes real knowledge, safest when it is merely weakly informative, and honest only when its influence is tested rather than hidden.