Model Drift
Model drift is the gradual loss of a deployed machine learning model's predictive quality because the world it was trained on has changed: the inputs it receives no longer look like its training data, or the relationship between inputs and the correct answer has itself moved. Nothing in the model's code or weights changes, which is exactly why the failure is quiet. The same frozen function keeps returning confident outputs while the reality they are meant to describe slides away from it.
Data drift and concept drift are different failures that need different responses. Data drift, also called covariate shift, means the distribution of the input features P(X) changes while the underlying rule linking inputs to outcomes stays put: a lending model trained on applicants from one region starts seeing a younger, higher-income population, or a camera is replaced and image brightness shifts. Label shift is a close cousin in which the proportion of classes changes, such as a fraud rate that suddenly doubles. Concept drift means P(Y given X) itself changes, so identical inputs now deserve different outputs: after a pricing change, the same browsing behaviour no longer predicts a purchase, and during the early pandemic, demand-forecasting models trained on years of normal patterns failed because the relationships themselves broke. Concept drift can be sudden, gradual, incremental or recurring and seasonal, a taxonomy set out in the survey by Gama and colleagues in 2014 on concept drift adaptation. Data drift does not always hurt accuracy, and concept drift can occur with perfectly stable inputs, so one cannot stand in for the other.
Detection starts with comparing distributions, and two statistics dominate practice. The Population Stability Index, long used in credit-risk scoring, bins a feature, then sums (actual share minus expected share) × ln(actual share divided by expected share) across bins; a common rule of thumb reads values under 0.1 as stable, 0.1 to 0.25 as moderate shift and above 0.25 as major, though these thresholds are convention rather than theorem. The two-sample Kolmogorov-Smirnov test measures the largest vertical gap between two empirical cumulative distribution functions of a continuous feature and gives a p-value, with the known caveat that very large samples make trivial differences statistically significant. Other tools include the Jensen-Shannon divergence, Wasserstein distance, chi-squared tests for categories and classifier-based detectors that try to tell reference from live data. Open-source libraries such as Evidently, NannyML and Alibi Detect package these checks.
The hardest part is that true performance needs labels, and labels often arrive late or never. A loan default is known months later, and a recommendation's value may never be observed. Teams therefore watch proxies: input drift, prediction drift (a shifting score distribution), confidence, and any delayed ground truth once it lands. Methods that estimate performance without labels exist but rest on assumptions, chiefly that only the inputs moved. A drifted feature is also not automatically a bug in the model; it is as often a broken upstream pipeline, a renamed column or a unit change, so the first response is to inspect the data source, not to retrain.
Retraining is the standard remedy but must be deliberate. Options include scheduled retraining on a rolling window, trigger-based retraining when a monitor fires, online or incremental learning, and sample weighting toward recent data. Each has risks: retraining on data produced by the model's own earlier decisions creates feedback loops, a new model can be worse than the old one, and fully automatic loops can propagate corrupted data. New versions should be validated against a holdout and rolled out cautiously. Model monitoring is the broader operational discipline that surfaces drift among latency, errors and cost; drift is the specific statistical phenomenon being watched for.
Drift matters for large language model systems too, though it looks different. The model weights may be frozen, yet user queries shift, retrieved documents change, and a hosted model can be silently updated by its provider, so evaluation sets must be re-run on production samples, not assumed to remain valid.
Read model drift as the expiry date that every trained model carries: the world keeps moving after the weights freeze, and the real engineering task is to notice which kind of movement occurred, input or concept, before deciding to retrain.