Confident Learning is a framework for identifying and correcting label errors in datasets by estimating the joint distribution of noisy (observed) labels and true (latent) labels. Developed by Curtis Northcutt et al., it provides principled methods for finding mislabeled examples and cleaning datasets.

Core Idea

Confident Learning estimates a confident joint — a matrix representing the joint distribution between noisy labels and true labels. From this joint, you can:

• Identify which examples are likely mislabeled
• Estimate the noise rate for each class
• Correct labels or remove noisy examples

How It Works

• Step 1 — Get Out-of-Sample Predictions: Train a model with cross-validation to get predicted probabilities for each example that weren't used during that example's training.
• Step 2 — Estimate Confident Joint: For each example, use a per-class threshold to determine the model's "confident" prediction. Count how often confident predictions disagree with given labels.
• Step 3 — Rank by Confidence: Examples where the model is most confident that the given label is wrong are ranked highest as likely label errors.
• Step 4 — Clean: Remove or re-label identified errors and retrain.

Key Properties

• Exact Label Errors: Confident Learning identifies specific mislabeled examples rather than just estimating noise rates.
• No Hyperparameters: The per-class thresholds are automatically computed from model predictions.
• Model-Agnostic: Works with any classifier that produces probability estimates — random forests, neural networks, or gradient boosting.

Cleanlab Library

The open-source cleanlab Python package implements Confident Learning with a simple API:

from cleanlab import Datalab
lab = Datalab(data=dataset, label_name="label")
lab.find_issues(pred_probs=predicted_probabilities)
lab.report()

Impact

The original paper demonstrated that major datasets contain significant label errors: ~3.4% in ImageNet, ~5.8% in MNIST, and ~6% in CIFAR-10. Cleaning these errors and retraining improved model accuracy on corrected test sets.

Confident Learning has become a standard tool for data-centric AI, used in production data pipelines to maintain and improve dataset quality.