Imbalanced Data

Imbalanced data is a classification dataset in which some classes are far rarer than others, so that a model trained naively sees the common class almost all the time and the class that actually matters almost never. Fraud, disease screening, equipment failure, intrusion detection, and defect inspection all share the shape: a widely used credit-card fraud dataset released by researchers at Université Libre de Bruxelles holds 492 frauds among 284,807 transactions, about 0.17 percent. The difficulty is not that the data is unusual, but that standard training and standard evaluation both quietly assume that classes appear in comparable numbers.

The first and most important consequence is that accuracy becomes close to meaningless: a model that predicts "not fraud" for every transaction scores roughly 99.8 percent on that dataset while catching nothing. Evaluation therefore has to move to measures that look at the rare class directly: recall (the share of real positives found), precision (the share of alarms that are real), their harmonic mean the F1 score, and curves that trade them off. Saito and Rehmsmeier's 2015 paper "The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets" explains why the ROC curve can look excellent when a flood of false positives is hiding behind a huge pool of negatives, whereas the precision-recall curve exposes it. The metric should also reflect the real costs, since missing a fraud and annoying a customer rarely cost the same.

The remedies fall into three families: change the data, change the loss, or change the decision rule. At the data level, random oversampling duplicates minority rows, random undersampling discards majority rows, and SMOTE (Chawla, Bowyer, Hall, and Kegelmeyer, 2002) synthesizes new minority examples by interpolating between a minority point and one of its nearest minority neighbors; the imbalanced-learn library packages these methods. At the loss level, errors on the rare class are made more expensive, either through weights (covered in its own entry on class weighting) or through the focal loss that Lin and colleagues introduced in 2017 for the RetinaNet object detector, which down-weights the flood of easy negatives. At the decision level, a probabilistic classifier is trained as usual and the threshold for calling something positive is moved away from 0.5 to match the desired precision-recall trade-off, a lever that costs nothing and is frequently the best first move.

Resampling has a trap that invalidates results quietly: it must happen after the train-test split, and inside each cross-validation fold, never before. If minority rows are duplicated or synthesized first and the data is split afterwards, copies of the same example land on both sides of the split and the test score measures memory rather than generalization. The evaluation set should keep the true class ratio, because that is the ratio production will see. Stratified splitting is the companion precaution, ensuring each fold contains enough rare examples to be meaningful at all, which with only 20 positives in a dataset is far from guaranteed.

Resampling also changes what the predicted probabilities mean. A model trained on a dataset balanced to 50-50 reports a probability that reflects the artificial prior, so a score of 0.5 no longer corresponds to a 50 percent chance in the real world; if calibrated probabilities are needed for ranking, pricing, or risk, they have to be recalibrated afterwards, for example with Platt scaling or isotonic regression on an untouched validation set. A related misconception is that more rebalancing is always better; frequently a strong gradient-boosted model with a tuned threshold matches heavy resampling, and aggressive oversampling of a noisy minority class amplifies the noise.

The same issue appears well beyond tabular classification. Object detectors face a vast majority of background boxes, segmentation models face tiny lesions inside large scans, and language models face long-tailed vocabularies and topics, where the rare cases can be the ones that matter for safety or fairness. When imbalance coincides with distribution shift, a model may also learn shortcuts that happen to track the majority class, so evaluating performance per slice or per subgroup is a necessary habit rather than a refinement.

Read imbalanced data as a warning that the usual defaults — accuracy as the score, an even treatment of every example, and a 0.5 cutoff — were all silently designed for balanced classes: the fix is to measure what matters on an honestly distributed test set, and only then reach for resampling, weighting, or threshold movement.