pseudo-labeling
**Pseudo-Labeling** is a **semi-supervised learning technique that leverages a small labeled dataset and a large unlabeled dataset** — training an initial model on the labeled data, using it to generate high-confidence predictions ("pseudo-labels") on the unlabeled data, then retraining on the combined labeled + pseudo-labeled data, effectively multiplying the training set size by 10-100× and achieving significant accuracy improvements when labeled data is scarce but unlabeled data is abundant.
**What Is Pseudo-Labeling?**
- **Definition**: A self-training approach where a model's own confident predictions on unlabeled data are treated as ground truth labels — creating a larger training set that combines real labels with model-generated "pseudo" labels for iterative improvement.
- **The Problem**: Labeled data is expensive (medical imaging: $10+ per label from radiologists, NLP: hours of annotation). But unlabeled data is cheap and abundant (millions of unlabeled images on the internet, billions of unlabeled text documents). How do you leverage this unlabeled data?
- **The Solution**: Train on the small labeled set, predict on the large unlabeled set, keep only the high-confidence predictions as pseudo-labels, retrain on everything. The model "teaches itself" from unlabeled data.
**The Pseudo-Labeling Process**
| Step | Process | Data Used |
|------|---------|-----------|
| 1. **Train teacher** | Train model on small labeled set | 1,000 labeled examples |
| 2. **Predict** | Apply teacher model to unlabeled data | 100,000 unlabeled examples |
| 3. **Filter** | Keep only predictions with confidence > threshold (e.g., 95%) | ~30,000 high-confidence pseudo-labels |
| 4. **Combine** | Merge real labels + pseudo-labels | 1,000 real + 30,000 pseudo = 31,000 |
| 5. **Retrain** | Train new model (student) on combined data | 31,000 training examples |
| 6. **Iterate** | Repeat with the improved student model | Progressive improvement |
**Confidence Threshold Impact**
| Threshold | Pseudo-Labels Generated | Quality | Accuracy Impact |
|-----------|----------------------|---------|----|
| **99%** | Few (conservative) | Very high quality, almost no noise | Small improvement (limited data added) |
| **95%** | Moderate | High quality with rare errors | Best balance (typical choice) |
| **90%** | Many | More noise introduced | Diminishing returns |
| **80%** | Very many | Significant noise | Can degrade performance (confirmation bias) |
| **50%** | Almost all data | Half are wrong | Model collapse (teaches itself garbage) |
**The Confirmation Bias Problem**
| Issue | Description | Mitigation |
|-------|------------|-----------|
| **Confirmation bias** | If the teacher is wrong and confident, it generates wrong pseudo-labels → student learns wrong patterns → cycle amplifies errors | High confidence threshold (>95%) |
| **Class imbalance amplification** | Model is more confident on majority class → pseudo-labels skew further toward majority | Class-balanced sampling, per-class thresholds |
| **Distribution shift** | Unlabeled data may have different distribution than labeled data | Domain adaptation techniques |
**Pseudo-Labeling vs Other Semi-Supervised Methods**
| Method | Approach | Pros | Cons |
|--------|---------|------|------|
| **Pseudo-Labeling** | Hard labels from confident predictions | Simple, framework-agnostic | Confirmation bias risk |
| **FixMatch** | Consistency regularization + pseudo-labels on strong augmentations | State-of-the-art accuracy | More complex implementation |
| **MixMatch** | Pseudo-labels + MixUp augmentation + consistency | Strong performance | Complex |
| **Self-Training** | Iterative pseudo-labeling (same idea, older name) | Simple | Same bias risk |
| **Co-Training** | Two models teach each other | Reduces single-model bias | Needs two views of data |
**Real-World Applications**
| Domain | Labeled Data | Unlabeled Data | Benefit |
|--------|-------------|---------------|---------|
| **Medical imaging** | 500 expert-labeled X-rays | 50,000 unlabeled X-rays | 10-15% accuracy improvement |
| **NLP classification** | 1,000 labeled reviews | 100,000 unlabeled reviews | Near-supervised-level performance |
| **Object detection** | 5,000 bounding boxes | 500,000 unlabeled images | Reduced annotation cost by 90% |
**Pseudo-Labeling is the simplest and most widely applicable semi-supervised technique** — enabling models to leverage vast amounts of unlabeled data by treating their own high-confidence predictions as training labels, effectively multiplying the labeled dataset size when annotation is expensive, with the critical requirement of a high confidence threshold to prevent the confirmation bias that can degrade model performance.