Sampling Bias
Sampling bias is a systematic distortion in which members of a population end up in a sample, causing the sample to misrepresent the population in a direction related to whatever is being measured — not random noise that would average out with more data, but a consistent skew that persists and even compounds as sample size grows. This is the specific failure mode that no amount of additional data collection can fix, which is what separates sampling bias from ordinary sampling error (the expected, random variation that comes from measuring a subset rather than the whole population): sampling error shrinks as a sample grows larger, while sampling bias does not, because the problem lies in who got selected, not in how many were selected.
Selection bias occurs whenever the mechanism used to choose a sample is correlated with the outcome being studied, and it shows up across a recognizable set of recurring patterns. Self-selection bias arises when individuals choose whether to participate, and that choice itself correlates with the trait being measured — customers who voluntarily leave product reviews tend to have unusually strong (positive or negative) opinions compared to the broader, mostly silent customer base, which is why online review scores are a well-documented example of a biased, not representative, sample of overall customer sentiment. Survivorship bias arises when a sample only includes entities that "survived" some selection process, systematically excluding the ones that didn't — studying only companies that are still in business today to draw conclusions about what makes a company successful silently excludes every company that failed for reasons that might have been just as instructive, and the pattern is frequently illustrated by the World War II statistician Abraham Wald's analysis of bomber damage: reinforcing the areas where returning aircraft showed bullet holes would have been a mistake, since the planes that didn't survive were likely hit in the areas showing no damage on the surviving sample — the aircraft that could still be observed were, by definition, not the ones hit in the most critical spots.
Undercoverage bias occurs when the sampling frame — the actual list or mechanism used to reach potential sample members — systematically excludes some portion of the true population, even unintentionally: a phone survey conducted only via landlines undercovers younger respondents who primarily or exclusively use mobile phones, and an online-only survey undercovers people without reliable internet access, both cases where the sampling frame itself, independent of any individual response choice, fails to give part of the population any chance of selection at all. This is exactly the mechanism behind the 1936 Literary Digest poll's famous failure: its sampling frame was drawn from telephone directories and car registration lists, which during the Depression systematically overrepresented wealthier households — a frame-level bias baked in before a single response was even collected, regardless of how large the resulting sample turned out to be.
Response bias, distinct from selection bias, occurs after a sample has already been correctly selected but respondents answer inaccurately in a systematic direction — social desirability bias, where respondents answer in a way they believe is more socially acceptable rather than strictly true (underreporting undesirable behaviors, overreporting desirable ones), is the most commonly cited example, and it's a distortion in the data collected rather than in who was selected to provide it, which is why fixing a biased sampling frame alone doesn't guarantee an unbiased result if the measurement instrument itself systematically distorts responses.
Mitigating sampling bias generally requires deliberately designing the sampling frame and selection process to be as representative as possible before data collection starts — using probability sampling methods with a genuinely comprehensive frame, actively checking a collected sample's demographic composition against known population figures, and being explicit about which population a sample's conclusions can and cannot be responsibly generalized to — since a biased sample, once collected, generally cannot be statistically repaired after the fact with any real confidence.
Read sampling bias as a structural flaw in who gets measured, not a statistical fluke that more data collection resolves: self-selection, survivorship, and undercoverage bias are the field's most commonly recurring patterns of this flaw, each illustrated by a well-documented historical case (biased product reviews, Abraham Wald's bomber analysis, the Literary Digest poll) precisely because the underlying mechanism keeps recurring in new forms, and the only genuine fix is addressing the selection process itself before data is collected, since a biased sample's distortion survives, and often worsens, no matter how much additional data is layered on top of it.