Label Smoothing is the regularization technique that replaces hard one-hot target labels with soft labels that distribute a small amount of probability mass to non-target classes — preventing the model from becoming overconfident in its predictions, improving calibration, and acting as an implicit regularizer that encourages the model to learn more generalizable representations rather than memorizing the exact training labels.

How Label Smoothing Works

• Hard label (standard): y = [0, 0, 1, 0, 0] (one-hot for class 2).
• Soft label (smoothing ε=0.1, K=5 classes): y = [0.02, 0.02, 0.92, 0.02, 0.02].
• Formula: $y_{smooth} = (1 - \varepsilon) \times y_{one-hot} + \varepsilon / K$
• Target class gets probability (1 - ε + ε/K), others get ε/K each.

Implementation

def label_smoothing_loss(logits, targets, epsilon=0.1):
    K = logits.size(-1)  # number of classes
    log_probs = F.log_softmax(logits, dim=-1)
    # NLL loss for true class
    nll = -log_probs.gather(dim=-1, index=targets.unsqueeze(1)).squeeze(1)
    # Uniform loss (smooth part)
    smooth = -log_probs.mean(dim=-1)
    loss = (1 - epsilon) * nll + epsilon * smooth
    return loss.mean()

Why Label Smoothing Helps

Typical ε Values

Relationship to Other Techniques

• Knowledge distillation: Teacher's soft predictions serve as implicit label smoothing.
• Mixup/CutMix: Create soft labels by mixing examples → similar regularization effect.
• Temperature scaling: Can be applied post-training for calibration (label smoothing does it during training).

When NOT to Use Label Smoothing

• When exact probabilities matter (some ranking/retrieval tasks).
• When combined with knowledge distillation (redundant smoothing).
• When label noise is already high (smoothing adds more uncertainty).

Label smoothing is one of the simplest and most effective regularization techniques available — adding just one hyperparameter (ε) that consistently improves generalization and calibration across vision, language, and speech models, making it a default inclusion in most modern training recipes.