noisy labels learning

**Noisy labels learning** (also called **learning from noisy labels** or **robust training**) encompasses machine learning techniques designed to train accurate models **despite errors in the training labels**. Since real-world datasets almost always contain some mislabeled examples, these methods are critical for practical ML. **Key Approaches** - **Robust Loss Functions**: Replace standard cross-entropy with losses that are less sensitive to mislabeled examples: - **Symmetric Cross-Entropy**: Combines standard CE with a reverse CE term. - **Generalized Cross-Entropy**: Interpolates between CE and mean absolute error. - **Truncated Loss**: Caps the loss for examples with very high loss (likely mislabeled). - **Sample Selection**: Identify and down-weight or remove likely mislabeled examples: - **Co-Teaching**: Train two networks simultaneously, each selecting "clean" examples for the other based on **small-loss criterion** — examples with high loss are likely mislabeled. - **Mentornet**: Use a separate "mentor" network to guide the main network's training by weighting examples. - **Confident Learning**: Estimate the **noise transition matrix** and use it to identify mislabeled examples. - **Regularization-Based**: Prevent the model from memorizing noisy labels: - **Mixup**: Blend training examples together, smoothing decision boundaries and reducing overfitting to noise. - **Early Stopping**: Stop training before the model starts memorizing noisy labels. - **Label Smoothing**: Soften hard labels to reduce the impact of any single mislabeled example. - **Noise Transition Models**: Explicitly model the probability of label corruption: - Learn a **noise transition matrix** T where $T_{ij}$ = probability that true class i is labeled as class j. - Use T to correct the loss function or the predictions. **When to Use** - **Large-Scale Web Data**: Datasets scraped from the internet invariably contain label errors. - **Distant Supervision**: Programmatically generated labels have systematic noise patterns. - **Crowdsourced Data**: Worker quality varies, producing noisy annotations. Noisy labels learning is an important practical concern — methods like **DivideMix** and **SELF** have shown that models can achieve **near-clean-data performance** even with **20–40% label noise**.