semi-supervised learning pseudo-labeling consistency regularization
# Semi-Supervised Learning: Pseudo-Labeling & Consistency Regularization
## Introduction & Motivation
Semi-supervised learning: leverage unlabeled data. Pseudo-labeling: self-train with high-confidence predictions. Consistency regularization: predictions invariant to perturbations. MixMatch: mix labeled and unlabeled data. FixMatch: weak and strong augmentation. Applications: limited labels, scalable learning, cost reduction.
Motivation: Unlabeled abundant, labeled scarce. Leverage unlabeled via self-supervision.
Applications: Limited labels, scaling to data, cost reduction.
---
## Core Concepts & Theory
### Pseudo-Labeling
High-confidence predictions as labels; self-train.
### Consistency Regularization
Perturbed samples should have same prediction.
### Entropy Minimization
Reduce prediction entropy; confident predictions.
---
## Mathematical Formulation
Pseudo-label loss:
$$L = L_{ ext{labeled}} + \lambda L_{ ext{pseudo}}$$
where L_pseudo on high-confidence predictions.
Consistency regularization:
$$L = \sum_i L(f(x_i), y_i) + \lambda \sum_j L_{ ext{consistency}}(f(x_j), f( ilde{x}_j))$$
Confidence threshold:
$$ ext{use\_pseudo} = \max(p) > au$$
---
## Advanced Theory & Extensions
### MixMatch
Mix labeled + unlabeled; label smoothing.
### FixMatch
Weak augment for pseudo-label; strong augment for consistency.
### ReMixMatch
Distribution alignment; remixing.
---
## Computational Considerations
Unlabeled: O(forward) per sample; free labels.
Consistency: O(2·forward) for augmented + original.
Pseudo-label: O(forward) forward pass.
---
## Practical Implementation Strategies
### Confidence Threshold
Start high (0.95); gradually lower.
### Augmentation Strength
Weak for pseudo-label; strong for consistency.
### Ramp-Up Schedule
Gradually increase unlabeled weight.
---
## Benchmark Datasets & Evaluation
CIFAR-10: SSL standard; 250 labels benchmark.
STL-10: 1000 labels; SSL standard.
ImageNet: Limited labels; cost-effective training.
---
## Key Challenges & Limitations
### Pseudo-Label Noise
Wrong labels → error propagation.
### Hyperparameter Tuning
Confidence threshold, augmentation, loss weights.
### Class Imbalance
May amplify imbalance; careful thresholding.
---
## Hyperparameter Tuning
Confidence τ: 0.95 typical; decrease over time.
λ (unlabeled weight): 0.1-1.0; schedule ramp-up.
Augmentation strength: weak for pseudo, strong for consistency.
---
## Real-World Applications & Case Studies
Limited Labels: Few labeled per class; scale via SSL.
Medical: Scarce annotations; pseudo-label carefully.
Vision: ImageNet SSL; competitive with supervised.
---
## Integration with Other Methods
SSL + Active Learning → hybrid labeling strategy.
SSL + Contrastive → dual objectives.
---
## Summary & Key Takeaways
Semi-supervised learning via pseudo-labeling and consistency regularization leverages unlabeled data for improved performance with limited annotation.
Principles:
1. Pseudo-labeling: confident predictions as labels.
2. Consistency: invariance to perturbations.
3. Entropy minimization: confident predictions.
4. Threshold: select confident samples.
5. Schedule: gradually increase unlabeled loss.
---
---
## Appendix: Practical Labs
### Lab 1: Pseudo-Labeling
import numpy as np
def generate_pseudo_labels(model, X_unlabeled, confidence_threshold=0.95):
"""Generate pseudo-labels for high-confidence predictions"""
probs = model.predict_proba(X_unlabeled)
confidences = probs.max(axis=1)
# Select high-confidence
mask = confidences > confidence_threshold
pseudo_labels = probs[mask].argmax(axis=1)
return X_unlabeled[mask], pseudo_labels, mask
# Test
np.random.seed(42)
class DummyModel:
def predict_proba(self, X):
return np.random.dirichlet([1]*10, len(X))
model = DummyModel()
X_unlabeled = np.random.randn(100, 10)
X_pseudo, y_pseudo, mask = generate_pseudo_labels(model, X_unlabeled)
assert len(X_pseudo) <= 100, "Subset selected"
print("✓ Pseudo-labeling working")
if __name__ == "__main__":
print("Lab 1: PseudoLabels - PASSED")### Lab 2: Consistency Loss
import torch
import torch.nn.functional as F
import numpy as np
def consistency_loss(model, X, X_perturbed):
"""Consistency regularization loss"""
pred_original = model(X)
pred_perturbed = model(X_perturbed)
# KL divergence
loss = F.kl_div(
F.log_softmax(pred_perturbed, dim=1),
F.softmax(pred_original.detach(), dim=1),
reduction='mean'
)
return loss
# Test
np.random.seed(42)
model = torch.nn.Linear(10, 5)
X = torch.randn(8, 10)
X_pert = X + 0.1 * torch.randn_like(X)
loss = consistency_loss(model, X, X_pert)
assert torch.isfinite(loss), "Loss finite"
print("✓ Consistency loss working")
if __name__ == "__main__":
print("Lab 2: Consistency - PASSED")### Lab 3: MixMatch
import numpy as np
def mixmatch(X_labeled, y_labeled, X_unlabeled, y_pseudo, alpha=0.75):
"""MixMatch: mix labeled and pseudo-labeled data"""
# Combine labeled and pseudo-labeled
X_combined = np.vstack([X_labeled, X_unlabeled])
y_combined = np.hstack([y_labeled, y_pseudo])
# Mixup
lam = np.random.beta(alpha, alpha)
idx = np.random.permutation(len(X_combined))
X_shuffled = X_combined[idx]
y_shuffled = y_combined[idx]
X_mix = lam * X_combined + (1 - lam) * X_shuffled
y_mix = lam * y_combined + (1 - lam) * y_shuffled
return X_mix, y_mix
# Test
np.random.seed(42)
X_l = np.random.randn(20, 10)
y_l = np.random.randint(0, 5, 20)
X_u = np.random.randn(50, 10)
y_u = np.random.randint(0, 5, 50)
X_mix, y_mix = mixmatch(X_l, y_l, X_u, y_u)
assert X_mix.shape == (70, 10), "Shape correct"
print("✓ MixMatch working")
if __name__ == "__main__":
print("Lab 3: MixMatch - PASSED")### Lab 4: SSL Training Loop
import numpy as np
def ssl_training_step(model, X_labeled, y_labeled, X_unlabeled, confidence_threshold=0.95):
"""Single SSL training step"""
# Pseudo-label
probs = model.predict_proba(X_unlabeled)
confidences = probs.max(axis=1)
mask = confidences > confidence_threshold
X_pseudo = X_unlabeled[mask]
y_pseudo = probs[mask].argmax(axis=1)
# Combine
X_combined = np.vstack([X_labeled, X_pseudo])
y_combined = np.hstack([y_labeled, y_pseudo])
# Train (simulated)
loss = (y_combined - y_combined).mean()
return loss
# Test
np.random.seed(42)
class DummyModel:
def predict_proba(self, X):
return np.random.dirichlet([1]*10, len(X))
model = DummyModel()
X_l = np.random.randn(20, 10)
y_l = np.random.randint(0, 10, 20)
X_u = np.random.randn(50, 10)
loss = ssl_training_step(model, X_l, y_l, X_u)
assert np.isfinite(loss), "Loss finite"
print("✓ SSL training working")
if __name__ == "__main__":
print("Lab 4: Training - PASSED")