confidence penalty

**Confidence Penalty** is a **regularization technique that penalizes the model for making overconfident predictions** — adding a penalty term to the loss that discourages the model from outputting predictions with very low entropy (highly concentrated probability distributions). **Confidence Penalty Formulation** - **Penalty**: $L = L_{task} - eta H(p)$ where $H(p) = -sum_c p(c) log p(c)$ is the entropy of the predicted distribution. - **Effect**: Maximizing entropy encourages spreading probability across classes — prevents overconfidence. - **$eta$ Parameter**: Controls the penalty strength — larger $eta$ = more uniform predictions. - **Relation**: Equivalent to label smoothing with a uniform target distribution. **Why It Matters** - **Calibration**: Overconfident models are poorly calibrated — confidence penalty improves calibration. - **Exploration**: In active learning and RL, confidence penalty encourages exploration of uncertain regions. - **Distillation**: Better-calibrated teacher models produce more informative soft labels for distillation. **Confidence Penalty** is **punishing overconfidence** — explicitly penalizing low-entropy predictions to produce better-calibrated, more honest models.