Cost-Sensitive Learning Asymmetric Loss Decision Theory
# Cost-Sensitive Learning: Asymmetric Loss & Decision Theory
## Introduction & Motivation
Cost-sensitive learning assigns different costs to different misclassification types. FP (false positive) cost ≠ FN (false negative) cost. Minimizes expected cost rather than error rate. Naturally handles imbalanced, high-stakes domains. Decision theory: classify if E[cost|class=1] < E[cost|class=0].
Motivation: Simple accuracy ignores asymmetric consequences. Cancer diagnosis: missing positive (FN) more harmful than false alarm (FP).
Applications: Medical diagnosis, fraud detection, loan approval, content moderation.
---
## Core Concepts & Theory
### Cost Matrix
C_FP = cost of false positive; C_FN = cost of false negative; C_TP = C_TN = 0.
### Expected Cost
E[cost] = C_FN × P(y=0|x) × P(predict 1) + C_FP × P(y=1|x) × P(predict 0).
### Cost-Sensitive Threshold
Classify positive if P(y=1|x) > τ where τ = C_FP / (C_FP + C_FN).
---
## Mathematical Formulation
Cost-sensitive loss:
$$\mathcal{L}_{ ext{cost}} = C_{FN} \cdot \mathbb{I}[\hat{y} = 0, y = 1] + C_{FP} \cdot \mathbb{I}[\hat{y} = 1, y = 0]$$
Expected risk:
$$R(\hat{y}) = \mathbb{E}_{(x,y)}[\mathcal{L}_{ ext{cost}}(\hat{y}, y)]$$
Optimal decision:
$$\hat{y}(x) = \arg\max_c [-C(c|y)] P(y|x)$$
---
## Advanced Theory & Extensions
### Weighted Loss Functions
Weighted cross-entropy in neural networks: w_c × L_c.
### Reject Option
Abstain on uncertain cases; return cost of rejection vs. classification.
### Cost Learning
Learn cost matrix from data via meta-learning.
---
## Computational Considerations
Cost matrix: O(k²) storage for k classes; O(k) per prediction.
Threshold search: O(n) to evaluate costs across thresholds.
Training: O(epochs × n) with cost-weighted gradients.
---
## Practical Implementation Strategies
### Domain Expert Input
Elicit costs from stakeholders; often domain-dependent.
### Normalization
Normalize cost matrix to [0,1]; improves numerical stability.
### Sensitivity Analysis
Evaluate performance across cost ranges; robustness.
---
## Benchmark Datasets & Evaluation
Medical: UCI Medical datasets; C_FN >> C_FP (missing disease).
Finance: Credit/loan datasets; C_FP (type I, bad loan approval) vs. C_FN (type II, reject good).
Metrics: Cost, expected utility, cost-sensitive accuracy.
---
## Key Challenges & Limitations
### Cost Elicitation
Hard to specify accurate costs; often domain knowledge required.
### Cost Distribution Shift
Costs change over time; requires periodic updates.
### Multi-Class Costs
k classes → k² costs; specification complex.
---
## Hyperparameter Tuning
C_FN / C_FP ratio: 1-100; domain-dependent.
Threshold: Derive from cost ratio; verify empirically.
Loss weight: Iterate with domain experts.
---
## Real-World Applications & Case Studies
Lending: Cost of bad loan >> opportunity cost of rejecting good applicant.
Medical: Cost of false negative (missing cancer) >> false positive (repeat test).
Content Moderation: Type I (remove valid) vs. Type II (allow invalid).
---
## Integration with Other Methods
Cost-Sensitive + Imbalanced → SMOTE + cost weights.
Cost-Sensitive + RL → reward is negative cost; maximize utility.
---
## Summary & Key Takeaways
Cost-sensitive learning explicitly minimizes expected asymmetric cost via weighted loss and decision-threshold optimization.
Principles:
1. Cost matrix specifies misclassification consequences.
2. Expected cost objective replaces standard loss.
3. Optimal threshold derived from cost ratio.
4. Cost elicitation requires domain expertise.
5. Sensitivity analysis evaluates robustness.
---
---
## Appendix: Practical Labs
### Lab 1: Cost Matrix & Expected Cost
import numpy as np
def compute_expected_cost(y_true, y_pred_proba, cost_matrix):
"""
cost_matrix[i,j] = cost of predicting i when true is j
"""
costs = []
for i, prob_pos in enumerate(y_pred_proba):
# Cost if predict 0 (predict negative)
cost_neg = cost_matrix[0, y_true[i]] * (1 - prob_pos)
# Cost if predict 1 (predict positive)
cost_pos = cost_matrix[1, y_true[i]] * prob_pos
costs.append(min(cost_neg, cost_pos))
return np.mean(costs)
# Data
np.random.seed(42)
y_true = np.array([0, 0, 1, 1, 0, 1, 0, 1])
y_pred_proba = np.array([0.1, 0.2, 0.8, 0.9, 0.3, 0.7, 0.15, 0.85])
# Cost matrix: FP cost = 1, FN cost = 10
cost_matrix = np.array([[0, 10], # predicting 0
[1, 0]]) # predicting 1
expected_cost = compute_expected_cost(y_true, y_pred_proba, cost_matrix)
print(f"Expected cost: {expected_cost:.2f}")
assert expected_cost >= 0, "Cost should be non-negative"
print("✓ Expected cost working")
if __name__ == "__main__":
print("Lab 1: Expected Cost - PASSED")### Lab 2: Cost-Sensitive Threshold
import numpy as np
def cost_sensitive_threshold(cost_fp, cost_fn):
"""Optimal threshold from cost ratio"""
return cost_fp / (cost_fp + cost_fn)
# Different cost ratios
cost_scenarios = [
(1, 10), # FN 10x worse than FP
(5, 5), # Equal costs
(10, 1) # FP 10x worse than FN
]
for cost_fp, cost_fn in cost_scenarios:
threshold = cost_sensitive_threshold(cost_fp, cost_fn)
print(f"Cost FP={cost_fp}, FN={cost_fn} -> Threshold={threshold:.3f}")
assert 0 <= threshold <= 1, "Threshold should be in [0,1]"
print("✓ Cost-sensitive threshold working")
if __name__ == "__main__":
print("Lab 2: Threshold - PASSED")### Lab 3: Cost-Sensitive Classification
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
# Data
np.random.seed(42)
X = np.random.randn(200, 5)
y = np.concatenate([np.zeros(150), np.ones(50)])
model = LogisticRegression(max_iter=200, random_state=42)
model.fit(X, y)
y_proba = model.predict_proba(X)[:, 1]
# Cost-sensitive classification
cost_fp, cost_fn = 1, 5 # FN is 5x worse
threshold = cost_fp / (cost_fp + cost_fn)
y_pred = (y_proba >= threshold).astype(int)
cm = confusion_matrix(y, y_pred)
tn, fp, fn, tp = cm.ravel()
total_cost = fp * cost_fp + fn * cost_fn
print(f"Threshold: {threshold:.3f}")
print(f"TP={tp}, FP={fp}, FN={fn}, TN={tn}")
print(f"Total cost: {total_cost}")
assert total_cost >= 0, "Cost should be non-negative"
print("✓ Cost-sensitive classification working")
if __name__ == "__main__":
print("Lab 3: Classification - PASSED")### Lab 4: ROC Curve with Cost
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_curve, auc
# Data
np.random.seed(42)
X = np.random.randn(300, 5)
y = np.concatenate([np.zeros(250), np.ones(50)])
model = LogisticRegression(max_iter=200, random_state=42)
model.fit(X, y)
y_proba = model.predict_proba(X)[:, 1]
# ROC curve
fpr, tpr, thresholds = roc_curve(y, y_proba)
roc_auc = auc(fpr, tpr)
# Cost-based operating point
cost_fp, cost_fn = 1, 5
cost_threshold = cost_fp / (cost_fp + cost_fn)
cost_idx = np.argmin(np.abs(thresholds - cost_threshold))
print(f"ROC AUC: {roc_auc:.4f}")
print(f"Cost-based operating point at threshold {thresholds[cost_idx]:.3f}")
print(f" FPR: {fpr[cost_idx]:.3f}, TPR: {tpr[cost_idx]:.3f}")
assert 0 <= roc_auc <= 1, "AUC should be in [0,1]"
assert 0 <= fpr[cost_idx] <= 1, "FPR should be in [0,1]"
print("✓ ROC with cost working")
if __name__ == "__main__":
print("Lab 4: ROC Cost - PASSED")