selective prediction
**Selective Prediction** is a machine learning framework where the model has the option to abstain from making predictions on inputs where it is insufficiently confident, trading coverage (fraction of inputs receiving predictions) for improved accuracy on the predictions it does make. By declining to predict on difficult or ambiguous inputs, selective prediction systems achieve higher reliability on their accepted predictions while flagging uncertain cases for human review.
**Why Selective Prediction Matters in AI/ML:**
Selective prediction enables **deployment of imperfect models in high-stakes applications** by ensuring that when the model does make a prediction, it meets a minimum reliability threshold, while uncertain cases are escalated rather than decided incorrectly.
• **Risk-coverage tradeoff** — Selective prediction creates a parameterizable tradeoff: at high coverage (predicting on most inputs) accuracy approaches the base model; at low coverage (predicting only on high-confidence inputs) accuracy approaches 100%; the risk-coverage curve characterizes this tradeoff
• **Selection function** — A selection function g(x) ∈ {0,1} decides whether to predict or abstain for each input; common implementations threshold the model's confidence score, uncertainty estimate, or a separately trained selector
• **Selective accuracy** — Performance is measured by selective accuracy (accuracy on accepted predictions), coverage (fraction of inputs receiving predictions), and the Area Under the Risk-Coverage curve (AURC) which summarizes the full tradeoff
• **Human-AI collaboration** — Selective prediction naturally implements human-in-the-loop systems: the model handles routine, high-confidence cases automatically while routing uncertain cases to human experts, optimizing overall system performance
• **Calibration dependency** — Selective prediction effectiveness depends heavily on calibration quality: a well-calibrated model's confidence scores reliably distinguish easy from hard inputs, while a miscalibrated model may abstain on easy cases and predict on hard ones
| Configuration | Coverage | Selective Accuracy | Use Case |
|--------------|----------|-------------------|----------|
| No Selection | 100% | Base model accuracy | Standard deployment |
| Low Threshold | 90-95% | +1-3% above base | Minor improvement |
| Medium Threshold | 70-85% | +5-10% above base | Balanced operation |
| High Threshold | 40-60% | +15-25% above base | Safety-critical |
| Expert Cascade | Variable | Near-expert level | Medical, legal |
**Selective prediction transforms AI deployment from an all-or-nothing proposition into a calibrated confidence-aware system that provides reliable predictions when confident and appropriately escalates uncertain cases, enabling the safe use of imperfect models in high-stakes applications through principled abstention rather than unreliable guessing.**