Dispersion

Dispersion is the statistical property of how widely the values of a dataset or distribution are spread out, and the measures of dispersion, also called measures of spread or variability, are the numbers that quantify it. Two classes can have the same average exam score of 70, yet in one every student scored between 65 and 75 while in the other scores ran from 30 to 100; central tendency cannot tell these apart, and dispersion is the second essential half of any description that can. The concept is the question of how much values differ from one another, and the individual measures are different answers to it.

Measures of dispersion fall into a few families, distinguished by what they use from the data, and each trades simplicity against robustness. The range, the difference between the maximum and minimum, is the simplest but depends on just two values and grows with sample size. The interquartile range, the distance between the 25th and 75th percentiles, ignores the extremes and captures the spread of the middle half. The variance averages squared distances from the mean, and the standard deviation, its square root, returns that to the original units; the mean absolute deviation and median absolute deviation average or take the median of absolute distances and are gentler on outliers. The term standard deviation was introduced by Karl Pearson in 1893, and Ronald Fisher introduced the word variance in 1918, when he developed its decomposition into components in the analysis of variance.

The choice of measure should follow the centre it accompanies and the shape of the data: standard deviation goes with the mean, the interquartile range or median absolute deviation with the median. For roughly symmetric, outlier-free data the standard deviation is efficient and has the rich interpretation of the empirical rule. For skewed data, or data with contamination, a single extreme value can inflate the standard deviation and range dramatically, while barely changing the interquartile range. Dispersion also depends on the scale of measurement, so absolute measures cannot be compared across variables with different units or very different means, which is the reason for the relative measure below.

For comparing variability across different scales, the coefficient of variation, the standard deviation divided by the mean and often shown as a percentage, expresses spread relative to size, a ratio sometimes credited to Karl Pearson in 1896. It tells whether a standard deviation of 5 grams is large (on a 20-gram sample) or trivial (on a 2,000-gram one), and it is used in laboratory assay precision and in comparing the volatility of assets of different price levels. It only makes sense for ratio-scale data with a true zero and breaks down when the mean is near zero. A different kind of dispersion measure, the Gini coefficient proposed by Corrado Gini in 1912, quantifies inequality in how a quantity such as income is distributed, and for categorical data, dispersion is measured by quantities such as entropy, since there is no distance between categories.

Dispersion is not a nuisance but often the main object of study. In manufacturing, the whole logic of quality control and of Six Sigma is the reduction of variation around a target, because a process with a lower standard deviation produces fewer defects. In finance, volatility, the standard deviation of returns, is the standard measure of risk. In statistical modelling, the spread of residuals measures model fit, the standard error is built from dispersion, and when count data are more dispersed than a Poisson model allows, with variance exceeding the mean, the problem called overdispersion requires models such as the negative binomial. In machine learning, features are standardised by their dispersion, and the bias-variance trade-off is a statement about the dispersion of predictions across training sets. In science, the spread of repeated measurements is the quantitative expression of experimental precision.

The recurring pitfalls are reporting a centre without any spread, assuming a given spread measure is comparable across data of different size or type, and reading dispersion as error. Small samples give unstable estimates of dispersion, and sample standard deviation underestimates the population value unless n minus 1 is used in the denominator for variance. Variation can also be real: a wide spread in patient responses may mean subgroups, not noise. And low dispersion is not automatically good, since it may reflect a measurement that lacks resolution or a population sampled too narrowly.

Read dispersion as the partner of central tendency: the number that says how far values wander from the centre, available in robust and non-robust forms, essential for judging whether a typical value is truly typical, and for knowing how much weight any single observation can bear.