Semi-Supervised Learning

Semi-supervised learning is the family of machine learning methods that train a model from a small set of labeled examples together with a much larger set of unlabeled ones, using the unlabeled data to learn the structure of the input space so that the few labels go further. It sits between supervised learning (every example has a label) and unsupervised learning (none do), and it matters because labels are usually the expensive part: a radiologist, lawyer or linguist must annotate each case, while raw images, documents and recordings are abundant and nearly free. The promise is a model that approaches the accuracy of a fully labeled dataset using only a fraction of the annotation budget.

Semi-supervised methods work only when the unlabeled data carries information about the labeling task, which is why the field is organized around a few assumptions. The smoothness assumption says that nearby points in input space should share a label; the cluster (low-density separation) assumption says that classes form groups and the decision boundary should pass through sparse regions rather than cut a dense cluster in half; and the manifold assumption says that high-dimensional data lies near a low-dimensional surface along which labels vary smoothly. Chapelle, Schölkopf and Zien's 2006 book "Semi-Supervised Learning" lays these out, and it also records an uncomfortable fact: when the assumptions fail, unlabeled data can hurt, so the technique is not a free lunch.

The classical techniques are self-training, co-training and graph-based label propagation. Self-training, whose idea dates to the 1960s, has a model label its own most confident predictions on unlabeled data and retrain on them as pseudo-labels; co-training (Blum and Mitchell, 1998) uses two classifiers built on different, conditionally independent views of the data (a web page's own text versus the text of links pointing to it) that label examples for each other; and label propagation (Zhu and Ghahramani, 2002) builds a similarity graph over all examples and lets labels diffuse along its edges. Generative approaches also exist, such as the semi-supervised variational autoencoder of Kingma and colleagues (2014), and the transductive support vector machine pushes the margin through low-density regions of the unlabeled data.

The deep learning era merged these ideas into consistency regularization: a model should give the same prediction for an unlabeled input even after it is perturbed. The Pi-model and temporal ensembling (Laine and Aila, 2017) and Mean Teacher (Tarvainen and Valpola, 2017) enforced agreement between differently perturbed or differently averaged copies of a network; MixMatch (Berthelot and colleagues, 2019) combined consistency, entropy minimization and the MixUp interpolation trick; and FixMatch (Sohn and colleagues, 2020) simplified the recipe to pseudo-labeling from a weakly augmented image, kept only above a confidence threshold, and trained as the target for a strongly augmented version of the same image. The authors reported that FixMatch reached roughly 94 percent accuracy on CIFAR-10 with only 250 labeled images, a result that made semi-supervised learning look nearly as good as full labeling on that benchmark. Noisy Student (Xie and colleagues, 2019) scaled self-training to ImageNet, using an EfficientNet teacher to label 300 million unlabeled images for a larger, noised student.

The central pitfall is confirmation bias: a model that trains on its own mistakes becomes more certain of them. If the first classifier is wrong about a region of input space, the pseudo-labels propagate the error and the confidence filter only cements it; class imbalance makes this worse, because a classifier that over-predicts the majority class floods the training set with majority labels. Another trap is distribution mismatch: if the unlabeled pool contains classes absent from the labeled set, performance can drop below that of using labels alone, as Oliver and colleagues' 2018 paper "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms" found, and it also criticized evaluating on tuned validation sets far larger than any realistic labeled budget. Honest evaluation requires a small validation set drawn like the labeled data and a comparison against a strong supervised baseline using the same labels.

The term is often confused with its neighbors, and the boundaries are worth stating. Self-supervised learning, as used to pretrain BERT or GPT, invents labels from the data itself (mask a word, predict the next one) and needs no labeled data at all in its first phase; semi-supervised learning mixes real labels with unlabeled data in one objective, although in practice the two are often combined by pretraining self-supervised then fine-tuning with pseudo-labels. Weak supervision, which uses noisy labeling rules, and active learning, which chooses which examples a human should label next, are different strategies for the same shortage of labels. In current practice, large pretrained models have absorbed much of the benefit that semi-supervised methods once supplied, yet pseudo-labeling and distillation remain common whenever a team has a trained model and an unlabeled backlog.

Read semi-supervised learning as letting a few trusted labels steer a sea of unlabeled data: its techniques differ in how they exploit that data, but all rest on the bet that nearby or consistent examples share labels, and all share the same risk, that confident mistakes feed on themselves.