Weakly Supervised Learning Noisy Partial Label Learning
# Weakly Supervised Learning: Noisy & Partial Label Learning
## Introduction & Motivation
Weakly supervised: incomplete, inaccurate, or imprecise annotations. Noisy labels: mislabeled examples; label smoothing, robust losses. Partial labels: subset of possible labels; learn label distributions. Multiple instance learning (MIL): bag-level only; instance-level unknown. Applications: crowdsourcing, medical diagnosis, large-scale data.
Motivation: Labeling expensive/error-prone. Weak supervision leverages imperfect labels.
Applications: Crowdsourcing, medical, weakly annotated data.
---
## Core Concepts & Theory
### Label Noise
Incorrect labels; training despite errors.
### Partial Labels
Candidate label set; true label subset.
### Multiple Instance Learning
Bag labels; instance labels unknown.
---
## Mathematical Formulation
Noisy label learning:
$$L = \sum_i l(y_i, \hat{y}_i) ext{ where } y_i ext{ may be wrong}$$
Robust loss (MAE):
$$L_{ ext{MAE}} = |y_i - \hat{y}_i|$$
Partial label loss:
$$L = -\sum_{c \in S_i} \log p(y=c|x_i)$$
where S_i = candidate label set.
---
## Advanced Theory & Extensions
### Crowdsourcing
Multiple annotators; aggregate labels.
### Self-Training
Use confident predictions; progressive refinement.
### Co-Training
Multiple views; train separately, advise each other.
---
## Computational Considerations
Noise robust: Same complexity as standard.
MIL: O(bag_size × instances) aggregation.
Crowdsourcing: O(annotators × samples) label processing.
---
## Practical Implementation Strategies
### Label Smoothing
Mix true label with uniform; soften targets.
### Confident Learning
Identify mislabeled; filter or reweight.
### Meta-Learning
Learn label noise correction; Kaggle solutions.
---
## Benchmark Datasets & Evaluation
Clothing1M: Real-world noisy labels; 1M images.
WebVision: Web images; inherent noise.
Medical: Crowdsourced annotations; aggregation.
---
## Key Challenges & Limitations
### Label Noise Bias
Affects model; limits performance ceiling.
### Overfitting Noise
Model memorizes; generalization hurts.
### Annotation Entropy
How much noise tolerable.
---
## Hyperparameter Tuning
Label smoothing α: 0.1-0.2; regularization.
Noise rate: Empirical estimate; data-dependent.
Robust loss: Choose per application.
---
## Real-World Applications & Case Studies
Crowdsourcing: Amazon Mechanical Turk; label aggregation.
Medical Imaging: Multiple radiologists; consensus labels.
Web Data: User annotations; inherent noise.
---
## Integration with Other Methods
Weak Supervision + Active Learning → query hard, noisy.
Weak Supervision + Semi-Supervised → pseudo + noisy.
---
## Summary & Key Takeaways
Weakly supervised learning via robust losses, label smoothing, and noise-aware training leverages imperfect annotations for practical, scalable learning.
Principles:
1. Noisy labels: robust loss, label smoothing.
2. Partial: candidate label sets.
3. MIL: bag-level aggregation.
4. Crowdsourcing: multiple annotators.
5. Confident learning: filter mislabeled.
---
---
## Appendix: Practical Labs
### Lab 1: Label Smoothing
import torch
import torch.nn.functional as F
import numpy as np
def cross_entropy_with_label_smoothing(logits, labels, smoothing=0.1, num_classes=10):
"""Cross-entropy with label smoothing"""
# Standard one-hot
targets = F.one_hot(labels, num_classes).float()
# Smooth: mix with uniform
smoothed_targets = targets * (1 - smoothing) + smoothing / num_classes
# Cross-entropy on smoothed
loss = -torch.sum(smoothed_targets * F.log_softmax(logits, dim=1), dim=1).mean()
return loss
# Test
np.random.seed(42)
logits = torch.randn(32, 10)
labels = torch.randint(0, 10, (32,))
loss = cross_entropy_with_label_smoothing(logits, labels, smoothing=0.1)
assert torch.isfinite(loss), "Loss finite"
print("✓ Label smoothing working")
if __name__ == "__main__":
print("Lab 1: Smoothing - PASSED")### Lab 2: Robust Loss Functions
import torch
import torch.nn.functional as F
import numpy as np
def robust_losses(logits, labels):
"""Compare robust loss functions"""
ce_loss = F.cross_entropy(logits, labels)
mae_loss = torch.abs(logits.max(dim=1)[0] - labels.float()).mean()
return ce_loss, mae_loss
# Test
np.random.seed(42)
logits = torch.randn(32, 10)
labels = torch.randint(0, 10, (32,))
ce, mae = robust_losses(logits, labels)
assert torch.isfinite(ce) and torch.isfinite(mae), "Losses finite"
print("✓ Robust losses working")
if __name__ == "__main__":
print("Lab 2: RobustLoss - PASSED")### Lab 3: Partial Labels
import numpy as np
def partial_label_loss(probs, candidate_labels):
"""Loss for partial label learning"""
# Log probability of candidate labels
loss = 0
for i, cand in enumerate(candidate_labels):
if len(cand) > 0:
candidate_probs = probs[i, cand]
loss -= np.log(candidate_probs.sum() + 1e-8)
return loss / len(candidate_labels)
# Test
np.random.seed(42)
probs = np.random.dirichlet([1]*10, 32)
candidate_labels = [np.array([0, 1, 2]), np.array([3, 4, 5])] * 16
loss = partial_label_loss(probs, candidate_labels)
assert np.isfinite(loss), "Loss finite"
print("✓ Partial labels working")
if __name__ == "__main__":
print("Lab 3: PartialLabels - PASSED")### Lab 4: Label Noise Detection
import numpy as np
def detect_noisy_labels(y_true, y_pred, confidence_threshold=0.5):
"""Detect potentially mislabeled samples"""
# Confidence: max probability
confidences = np.max(y_pred, axis=1)
correct = (y_true == np.argmax(y_pred, axis=1))
# Low confidence correct = uncertain; high confidence wrong = mislabeled
mislabeled = (~correct) & (confidences > confidence_threshold)
return mislabeled
# Test
np.random.seed(42)
y_true = np.random.randint(0, 10, 100)
y_pred = np.random.dirichlet([1]*10, 100)
mislabeled = detect_noisy_labels(y_true, y_pred)
assert len(mislabeled) == 100, "Detection per sample"
print("✓ Noisy label detection working")
if __name__ == "__main__":
print("Lab 4: NoiseDetection - PASSED")