semi-supervised learning pseudo-labeling self-training
# Semi-Supervised Learning: Pseudo-Labeling & Self-Training
## Introduction & Motivation
Semi-Supervised Learning: leverage unlabeled data. Pseudo-labeling: assign labels to unlabeled samples. Self-training: iteratively improve. Applications: limited labels, medical imaging, NLP.
Motivation: Exploit unlabeled data; improve generalization.
Applications: Limited supervision, data-efficient learning.
---
## Core Concepts & Theory
### Pseudo-Labels
High-confidence predictions on unlabeled data.
### Consistency Regularization
Perturbed samples have same prediction.
### MixMatch and Variants
Combination of techniques; SOTA.
---
## Mathematical Formulation
Semi-supervised loss:
$$L = L_{ ext{labeled}} + \lambda L_{ ext{unlabeled}}$$
Pseudo-label loss:
$$L_{ ext{pseudo}} = \sum_u H(\hat{y}_u, f(x_u'))$$
where \hat{y}_u is pseudo-label, x_u' is augmented.
---
## Advanced Theory & Extensions
### MixMatch
Mix labeled and unlabeled; consistency.
### FixMatch
RandAugment + consistency regularization.
### ReMixMatch
Distribution alignment; better mixing.
---
## Computational Considerations
Pseudo-label: O(N) predictions.
Consistency: O(augmentation_cost).
Training: Modest overhead.
---
## Practical Implementation Strategies
### Confidence Threshold
Only use high-confidence pseudo-labels.
### Augmentation Strategy
Strong augmentation for unlabeled.
### Ramp-up Schedule
Gradually increase unlabeled weight.
---
## Benchmark Datasets & Evaluation
CIFAR-10/100 (limited): Semi-supervised benchmark.
STL-10: Semi-supervised benchmark.
SVHN: Digit recognition semi-supervised.
---
## Key Challenges & Limitations
### Pseudo-Label Errors
Incorrect high-confidence labels.
### Distribution Mismatch
Labeled vs. unlabeled data difference.
### Cold Start
Initial model quality; circular dependency.
---
## Hyperparameter Tuning
Confidence threshold: 0.7-0.95.
Unlabeled weight (λ): 0.1-1.0; ramp-up.
Augmentation: Task-specific strength.
---
## Real-World Applications & Case Studies
Medical Imaging: Scarce labels; abundant unlabeled.
NLP Classification: Few labeled documents.
Object Detection: Semi-supervised frameworks.
---
## Integration with Other Methods
Semi-supervised + Self-supervised → best practice.
Semi-supervised + Ensemble → stability.
---
## Summary & Key Takeaways
Semi-Supervised Learning via pseudo-labeling enables improved generalization through self-training on unlabeled data with consistency regularization.
Principles:
1. Pseudo-labels: high-confidence assignment.
2. Consistency: augmentation invariance.
3. Confidence threshold: quality control.
4. Ramp-up: gradual unlabeled weight.
5. Augmentation: critical for performance.
---
---
## Appendix: Practical Labs
### Lab 1: Pseudo-Label Generation
import numpy as np
def generate_pseudo_labels(predictions, confidence_threshold=0.9):
"""Generate pseudo-labels from predictions"""
confidence = np.max(predictions, axis=1)
predicted_labels = np.argmax(predictions, axis=1)
# Only keep high-confidence predictions
mask = confidence >= confidence_threshold
pseudo_labels = predicted_labels[mask]
valid_indices = np.where(mask)[0]
return pseudo_labels, valid_indices, confidence
# Test
np.random.seed(42)
preds = np.random.rand(100, 10)
preds = preds / preds.sum(axis=1, keepdims=True)
pseudo_labels, valid_idx, conf = generate_pseudo_labels(preds)
assert len(pseudo_labels) <= 100, "At most all samples"
assert len(pseudo_labels) == len(valid_idx), "Consistent indexing"
print("✓ Pseudo-label generation working")
if __name__ == "__main__":
print("Lab 1: PseudoLabelGeneration - PASSED")### Lab 2: Consistency Loss
import numpy as np
def consistency_loss(predictions_weak, predictions_strong):
"""Compute consistency loss between weak and strong augmentations"""
# KL divergence: weak as target, strong as prediction
kl_loss = np.sum(
predictions_weak * (np.log(predictions_weak + 1e-8) - np.log(predictions_strong + 1e-8)),
axis=1
).mean()
return kl_loss
# Test
np.random.seed(42)
pred_weak = np.random.rand(32, 10)
pred_weak = pred_weak / pred_weak.sum(axis=1, keepdims=True)
pred_strong = np.random.rand(32, 10)
pred_strong = pred_strong / pred_strong.sum(axis=1, keepdims=True)
loss = consistency_loss(pred_weak, pred_strong)
assert np.isfinite(loss), "Loss finite"
print("✓ Consistency loss working")
if __name__ == "__main__":
print("Lab 2: ConsistencyLoss - PASSED")### Lab 3: Semi-Supervised Loss
import numpy as np
def semi_supervised_loss(logits_labeled, targets_labeled, logits_unlabeled,
pseudo_labels, lambda_unsup=0.5):
"""Combine supervised and unsupervised losses"""
# Supervised loss
exp_logits = np.exp(logits_labeled - np.max(logits_labeled, axis=1, keepdims=True))
probs = exp_logits / exp_logits.sum(axis=1, keepdims=True)
sup_loss = -np.log(probs[np.arange(len(logits_labeled)), targets_labeled] + 1e-8).mean()
# Unsupervised loss
exp_logits_u = np.exp(logits_unlabeled - np.max(logits_unlabeled, axis=1, keepdims=True))
probs_u = exp_logits_u / exp_logits_u.sum(axis=1, keepdims=True)
unsup_loss = -np.log(probs_u[np.arange(len(pseudo_labels)), pseudo_labels] + 1e-8).mean()
# Combined
total_loss = sup_loss + lambda_unsup * unsup_loss
return total_loss
# Test
np.random.seed(42)
logits_l = np.random.randn(32, 10)
targets_l = np.random.randint(0, 10, 32)
logits_u = np.random.randn(32, 10)
pseudo = np.random.randint(0, 10, 32)
loss = semi_supervised_loss(logits_l, targets_l, logits_u, pseudo)
assert np.isfinite(loss), "Loss finite"
print("✓ Semi-supervised loss working")
if __name__ == "__main__":
print("Lab 3: SemiSupervisedLoss - PASSED")### Lab 4: Ramp-up Schedule
import numpy as np
def ramp_up_schedule(epoch, total_epochs, max_lambda=1.0, ramp_up_epochs=100):
"""Ramp-up schedule for unlabeled weight"""
if epoch < ramp_up_epochs:
# Cosine ramp-up
progress = epoch / ramp_up_epochs
lambda_unsup = max_lambda * (1 - np.cos(np.pi * progress)) / 2
else:
lambda_unsup = max_lambda
return lambda_unsup
# Test
lambdas = [ramp_up_schedule(e, 200) for e in range(200)]
assert lambdas[0] < lambdas[50], "Ramp-up increasing"
assert lambdas[-1] >= lambdas[-2], "Stable at end"
print("✓ Ramp-up schedule working")
if __name__ == "__main__":
print("Lab 4: RampUpSchedule - PASSED")