reject option
**Reject Option** is a formal decision-theoretic framework for classification where the model has three possible actions for each input: classify into one of the known classes, or reject (abstain from classification) when the expected cost of misclassification exceeds the cost of rejection. The reject option introduces an explicit cost for rejection (d) that is less than the cost of misclassification (c), creating an optimal rejection rule based on posterior class probabilities.
**Why Reject Option Matters in AI/ML:**
The reject option provides the **mathematical foundation for principled abstention**, defining exactly when a classifier should refuse to decide based on a formal cost analysis, rather than relying on ad-hoc confidence thresholds.
• **Chow's rule** — The optimal reject rule (Chow 1970) rejects input x when max_k p(y=k|x) < 1 - d/c, where d is the cost of rejection and c is the cost of misclassification; this minimizes the total expected cost (errors + rejections) and is provably optimal for known posteriors
• **Cost-based formulation** — The reject option formalizes the intuition that abstaining should be cheaper than guessing wrong: if misclassification costs $100 and human review costs $10, the model should reject whenever its confidence doesn't justify the $100 risk
• **Error-reject tradeoff** — Increasing the rejection threshold reduces error rate on accepted samples but increases the rejection rate; the error-reject curve characterizes this tradeoff, and the optimal operating point depends on the relative costs
• **Bounded improvement** — Theory shows that the reject option reduces the error rate on accepted samples from ε (base error) toward 0 as the rejection threshold increases, with the error-reject curve following a concave boundary determined by the Bayes-optimal classifier
• **Asymmetric costs** — In practice, different types of errors have different costs (false positive vs. false negative); the reject option extends to class-dependent costs with class-specific rejection thresholds, providing fine-grained control over which types of errors to avoid
| Component | Specification | Typical Value |
|-----------|--------------|---------------|
| Rejection Cost (d) | Cost of abstaining | $1-50 (application-dependent) |
| Misclassification Cost (c) | Cost of wrong prediction | $10-10,000 |
| Rejection Threshold | 1 - d/c | 0.5-0.99 |
| Error on Accepted | Error rate after rejection | Decreases with more rejection |
| Coverage | Fraction of accepted inputs | 1 - rejection rate |
| Optimal Rule | Chow's rule | max p(y=k|x) < threshold |
**The reject option provides the theoretically optimal framework for deciding when a classifier should abstain, grounding abstention decisions in formal cost analysis rather than arbitrary confidence thresholds, and establishing the mathematical foundation for all selective prediction systems that trade coverage for reliability in safety-critical AI applications.**