Confidence Score
A confidence score is a number a model attaches to a prediction to indicate how likely it believes that prediction is to be correct. Classifiers typically produce it as the probability assigned to the predicted class, language models can expose the probabilities of their output tokens, and many systems now also ask a model to state a confidence in words or as a percentage. The score is meant to support decisions such as accepting an answer automatically, routing it to a person, or abstaining.
The most common confidence score is the maximum softmax probability, the highest value in the vector the softmax function produces over classes. Hendrycks and Gimpel (2017) showed it works as a surprisingly strong baseline for spotting misclassified and out-of-distribution inputs: correct predictions tend to have higher maximum softmax values than incorrect ones. But the name is misleading. Softmax outputs are positive numbers that sum to one, which makes them look like probabilities, yet they are just a normalized transformation of the network's raw logits and carry no built-in guarantee about how often they are right.
The key caveat is that modern networks are overconfident, and softmax confidence can be high for inputs that are nonsense. Guo et al. (2017, "On Calibration of Modern Neural Networks") found that deep networks, unlike older shallower ones, tended to report confidence well above their actual accuracy, and that simple post-hoc fixes such as temperature scaling could repair much of the gap. Separately, Nguyen, Yosinski and Clune (2015) generated meaningless images that networks labelled with near-total confidence, and a model has no softmax option for "none of the above" unless one is added, since the outputs must sum to one over the classes it knows. Whether the scores match real accuracy is the subject of calibration, which is a separate entry and a separate property: a score can rank cases well (high for correct, low for incorrect) and still be badly calibrated.
Language models add two further complications: probabilities attach to tokens, not to facts, and post-training changes them. A sentence can be factually wrong while each token is individually likely, and a correct answer expressed in unusual wording gets a low sequence probability. Kadavath et al. (2022, "Language Models (Mostly) Know What They Know") found that large pretrained models were reasonably well calibrated on multiple-choice questions and could be trained to predict whether their own answers were correct. OpenAI's GPT-4 technical report (2023) showed a pretrained model's confidence tracking accuracy closely and the same measure degrading after RLHF post-training, a reminder that alignment training can distort the signal.
Verbalized confidence, where a model is asked to state how sure it is, is a distinct approach that works even through an API that hides probabilities. Lin, Hilton and Evans (2022, "Teaching Models to Express Their Uncertainty in Words") fine-tuned a model to output calibrated confidence phrases for arithmetic tasks, and Xiong et al. (2023) examined how well prompted language models express confidence, finding a tendency toward overconfidence, with stated values clustering at high round numbers such as 80 or 90 percent. Combining verbalized confidence with agreement across sampled answers generally gave better results than either alone. Such numbers should be treated as imperfect evidence, not as measured probabilities.
The pitfalls are practical and recurring. Confidence and correctness are different things: a model can be confidently wrong, especially on rare or adversarial inputs. A score threshold chosen on one dataset may not transfer to another, and users tend to read a displayed "95 percent" as a guarantee. The broader goal of knowing when to trust a model is covered by uncertainty-estimation, of which a confidence score is only one cheap signal.
Read a confidence score as the model's self-reported hunch rather than a measured probability: it is useful for ranking which answers to trust first, but only calibration against observed outcomes can tell you what a given number actually means.