Noisy labels learning (also called learning from noisy labels or robust training) encompasses machine learning techniques designed to train accurate models despite errors in the training labels. Since real-world datasets almost always contain some mislabeled examples, these methods are critical for practical ML.

Key Approaches

• Robust Loss Functions: Replace standard cross-entropy with losses that are less sensitive to mislabeled examples:
• Symmetric Cross-Entropy: Combines standard CE with a reverse CE term.
• Generalized Cross-Entropy: Interpolates between CE and mean absolute error.
• Truncated Loss: Caps the loss for examples with very high loss (likely mislabeled).

• Sample Selection: Identify and down-weight or remove likely mislabeled examples:
• Co-Teaching: Train two networks simultaneously, each selecting "clean" examples for the other based on small-loss criterion — examples with high loss are likely mislabeled.
• Mentornet: Use a separate "mentor" network to guide the main network's training by weighting examples.
• Confident Learning: Estimate the noise transition matrix and use it to identify mislabeled examples.

• Regularization-Based: Prevent the model from memorizing noisy labels:
• Mixup: Blend training examples together, smoothing decision boundaries and reducing overfitting to noise.
• Early Stopping: Stop training before the model starts memorizing noisy labels.
• Label Smoothing: Soften hard labels to reduce the impact of any single mislabeled example.

• Noise Transition Models: Explicitly model the probability of label corruption:
• Learn a noise transition matrix T where $T_{ij}$ = probability that true class i is labeled as class j.
• Use T to correct the loss function or the predictions.

When to Use

• Large-Scale Web Data: Datasets scraped from the internet invariably contain label errors.
• Distant Supervision: Programmatically generated labels have systematic noise patterns.
• Crowdsourced Data: Worker quality varies, producing noisy annotations.

Noisy labels learning is an important practical concern — methods like DivideMix and SELF have shown that models can achieve near-clean-data performance even with 20–40% label noise.