Labels

Labels are the correct answers attached to training data — the "cat" or "dog" tag on an image, the actual sale price next to a house's features, the next real word in a sentence — and they are what turn a pile of raw data into something a supervised model can actually learn from, since the model's entire training signal is the gap between its prediction and the label. Without labels, a model has nothing to be wrong about; this is the exact line that separates supervised learning (labels required) from unsupervised learning (no labels, the model finds structure on its own). At a high level, labels enter and drive the training loop like this:

Labels are expensive, and that cost is what shapes which problems get solved with supervised learning at all. ImageNet, the dataset that kicked off the deep-learning era, required roughly 14 million images hand-labeled across 20,000+ categories by crowdworkers over several years; medical-imaging labels are even more expensive because they require actual radiologists, not generic crowdworkers, to annotate tumors and anomalies. This cost asymmetry — data is nearly free to collect, labels are expensive to produce — is the direct reason self-supervised pretraining (where a model generates its own training signal from unlabeled text or images, as in masked-language-model pretraining for LLMs) became the dominant strategy for large models: it sidesteps the labeling bottleneck entirely for the pretraining phase, saving human annotation for a much smaller fine-tuning stage.

Label noise and label bias are two different failure modes with different fixes. Label noise is random — an annotator misclicks, or two radiologists disagree on a borderline case — and techniques like label smoothing (softening a hard 0/1 target to something like 0.9/0.1) and training with multiple annotators per example to average out disagreement both exist specifically to prevent a model from over-fitting to individual mislabeled points. Label bias is systematic — if a hiring dataset's historical labels reflect past discriminatory decisions, a model trained on them will learn and amplify that bias regardless of how clean the labeling process was, which is why auditing what a label actually measures matters as much as auditing labeling accuracy.

Weak labels and programmatic labeling trade label quality for label volume at a predictable rate. Snorkel-style weak supervision writes a set of heuristic labeling rules (keyword matches, pattern rules, existing noisy databases) and statistically combines their votes into probabilistic labels across millions of examples that would be far too expensive to hand-label — the resulting labels are noisier than expert annotation, but the sheer volume increase often more than compensates in final model accuracy, which is why most large industrial labeling pipelines today are hybrid: programmatic labels for scale, human review for the hardest or highest-stakes examples.

Read a label as the single most expensive bit of information in the entire training pipeline, byte for byte: a raw unlabeled image costs almost nothing to store on a hard drive, but the human or algorithmic judgment that turns it into a labeled example is what actually lets a model learn anything — the same reason foundries spend disproportionate effort on inline metrology and defect classification during fabrication, since a correctly labeled defect is what makes a wafer's yield data usable at all.