Bayesian Deep Learning and Uncertainty is the framework for quantifying model uncertainty through Bayesian inference — distinguishing epistemic (model) uncertainty from aleatoric (data) uncertainty to enable principled uncertainty estimation for safety-critical applications.
Uncertainty Decomposition:
- Epistemic uncertainty: model uncertainty; reducible with more training data; reflects uncertainty about parameters
- Aleatoric uncertainty: data/measurement uncertainty; irreducible; inherent noise in data generation process
- Total uncertainty: epistemic + aleatoric; total predictive uncertainty crucial for risk-aware decisions
- Heteroscedastic aleatoric: data-dependent noise level; different examples have different noise levels
Monte Carlo Dropout (Gal & Ghahramani):
- Bayesian interpretation: dropout can be interpreted as approximate Bayesian inference via variational inference
- MC sampling: perform multiple forward passes with dropout enabled (stochastic sampling from approximate posterior)
- Uncertainty quantification: variance across stochastic forward passes estimates model uncertainty
- Implementation: trivial modification to existing dropout networks; enable dropout at test time
- Computational cost: requires T forward passes (typically 10-50) per example; tradeoff between accuracy and computation
Deep Ensembles:
- Ensemble uncertainty: train multiple independent models (different initializations, hyperparameters, data subsets)
- Predictive mean: average predictions across ensemble; often better than single model
- Variance estimation: variance of predictions across ensemble estimates model uncertainty
- Aleatoric uncertainty: average predicted variance (if networks output variance) estimates aleatoric uncertainty
- Empirical strong baseline: surprisingly effective; often outperforms more complex Bayesian methods
- Ensemble disadvantage: computational cost proportional to ensemble size; multiple model storage
Laplace Approximation:
- Posterior approximation: approximate posterior as Gaussian around MAP solution; second-order Taylor expansion
- Hessian computation: curvature matrix (Fisher information) captures posterior uncertainty; computationally expensive
- Uncertainty from curvature: high curvature (confident) vs low curvature (uncertain) inferred from Hessian
- Scalability: Hessian computation challenging for large networks; various approximations (diagonal, KFAC) enable scalability
Calibration and Reliability:
- Model calibration: predicted confidence matches true accuracy; miscalibrated models overconfident/underconfident
- Expected calibration error (ECE): average difference between predicted confidence and actual accuracy; measures calibration
- Reliability diagrams: binned predictions showing confidence vs accuracy; visual assessment of calibration
- Temperature scaling: post-hoc calibration; adjust softmax temperature to achieve better calibration without retraining
- Calibration in deep networks: larger networks tend to be miscalibrated (overconfident); calibration essential for safety
Uncertainty Applications:
- Medical diagnosis: uncertainty guiding when to refer to specialist; clinical decision-making support
- Autonomous driving: uncertainty estimates enable collision avoidance; high-risk uncertainty triggers safety protocols
- Out-of-distribution detection: high epistemic uncertainty for OOD inputs; detect dataset shift and anomalies
- Active learning: select uncertain examples for labeling; efficient data annotation strategies
Safety-Critical Deployment:
- Risk-aware decisions: use uncertainty to abstain or request human intervention on high-uncertainty examples
- Confidence calibration: true uncertainty reflects decision quality; essential for safety-critical applications
- Uncertainty feedback: operator informed of model confidence; enables appropriate trust calibration
- Monitoring and drift detection: epistemic uncertainty changes indicate data distribution shift; triggers model retraining
Bayesian deep learning quantifies model and data uncertainty — enabling risk-aware decisions in safety-critical applications where understanding prediction confidence is essential for responsible deployment.