Probability
Probability is the mathematical framework for quantifying uncertainty, assigning a number between 0 and 1 to how likely an event is to occur, where 0 means the event is impossible and 1 means it's certain. It underlies essentially every inferential statistical method — confidence intervals, hypothesis tests, machine learning model outputs expressed as probabilities — because quantifying how confident a conclusion deserves to be requires a formal mathematical language for uncertainty itself, and probability theory is that language.
The field has historically supported two competing interpretations of what a probability actually means, a philosophical divide with real practical consequences for how statistics gets done: the frequentist interpretation defines probability as the long-run frequency of an event across infinitely many repeated trials (a coin's probability of landing heads is 0.5 because, over a very large number of flips, close to half would land heads), while the Bayesian interpretation defines probability as a degree of belief or confidence, which can be assigned even to one-off events that can't meaningfully be repeated (the probability that a specific historical event happened a certain way, or that a particular scientific hypothesis is true). This isn't a purely academic distinction: frequentist and Bayesian statistics use different mathematical machinery and produce different kinds of conclusions from the same data, and the choice between the two frameworks (classical hypothesis testing and p-values are frequentist tools; Bayesian inference updates a prior belief using observed data via Bayes' theorem) remains an active, substantive methodological divide across different subfields of statistics and data science.
Probability theory's modern axiomatic foundation comes from Andrey Kolmogorov's 1933 formalization, which established three basic axioms — probability is always non-negative, the probability of the entire sample space (all possible outcomes) equals 1, and the probability of mutually exclusive events happening together is the sum of their individual probabilities — from which essentially the entire mathematical structure of probability theory, both frequentist and Bayesian, can be rigorously derived. Before Kolmogorov's axiomatization, probability had developed piecemeal across several centuries through games of chance (Blaise Pascal and Pierre de Fermat's 17th-century correspondence on gambling problems is widely credited as an early formal starting point) without a single, unified, rigorous mathematical foundation underneath it.
A critical and frequently confused distinction runs through every application of probability: independence versus dependence between events. Two events are independent if the occurrence of one has no effect on the probability of the other (two separate coin flips), while events are dependent if one's occurrence changes the probability of the other (drawing a card from a deck without replacement changes the probabilities for the next draw, since the deck's composition has changed). Treating dependent events as though they were independent — a common statistical error, particularly with time-series or spatially correlated data — leads to systematically overconfident conclusions, understating true uncertainty, because the formulas for combining independent probabilities don't correctly account for the shared information dependent events carry.
Read probability as the formal language every statistical claim about uncertainty is ultimately written in: Kolmogorov's 1933 axioms are the rigorous mathematical foundation underlying both of the field's major interpretive camps, the frequentist-versus-Bayesian divide reflects a genuine, still-unresolved philosophical disagreement about what a probability actually represents rather than a settled technical matter, and correctly distinguishing independent from dependent events is one of the most basic but most consequential checks required before applying any probability calculation to real data.