label shift

**Label shift** (also called **prior probability shift** or **target shift**) is a type of distribution shift where the **distribution of output labels P(Y) changes** between training and deployment, while the class-conditional input distribution P(X|Y) remains the same. **Intuitive Example** - A **spam detector** is trained when 10% of emails are spam. At deployment, spam increases to 40%. The characteristics of spam and non-spam emails haven't changed — but their **proportions** have shifted. - A **disease classifier** trained on hospital data where 2% of patients have the disease, deployed in a screening program where 15% have it. **Why Label Shift Matters** - Models implicitly learn **class prior probabilities** from training data. If the prior changes, the model's calibration and decision boundaries become suboptimal. - **Precision and recall** are affected — a model tuned for rare positives will under-predict when positives become more common. - **Threshold-based decisions** break — the optimal classification threshold depends on class priors. **Detection** - **Monitor Class Proportions**: Track the distribution of predicted classes over time. Significant changes in prediction proportions may indicate label shift. - **Black Box Shift Detection (BBSD)**: Use model predictions to estimate whether the label distribution has changed. - **Confusion Matrix Monitoring**: Track precision, recall, and other metrics across time windows. **Correction Methods** - **Importance Weighting**: Re-weight training examples based on the ratio of target-to-source class proportions. If class A is 2× more common in deployment, upweight class A training examples by 2×. - **Expectation Maximization**: Iteratively estimate the new class prior and adjust the model's outputs accordingly. - **Threshold Adjustment**: Modify the classification threshold to account for the new class balance without retraining. - **Calibration**: Re-calibrate model probabilities on data representative of the deployment distribution. **Label Shift vs. Other Shifts** - **Covariate Shift**: Input P(X) changes, P(Y|X) stays same. - **Label Shift**: Output P(Y) changes, P(X|Y) stays same. - **Concept Drift**: P(Y|X) itself changes — fundamentally different and harder to handle. Label shift is one of the **simpler** forms of distribution shift to correct because the fundamental input-output relationship hasn't changed — only the proportions have.