bayesian
**Bayesian Deep Learning** is the **framework that treats neural network weights as probability distributions rather than fixed values** — enabling principled uncertainty quantification by maintaining a posterior distribution over all possible model parameters, producing predictions that account for both aleatoric uncertainty in data and epistemic uncertainty from limited training.
**What Is Bayesian Deep Learning?**
- **Definition**: Apply Bayesian inference to neural networks — instead of finding a single optimal weight vector θ* via maximum likelihood, maintain a posterior distribution P(θ|data) over all possible weight configurations and integrate over this distribution to make predictions.
- **Standard Deep Learning**: θ* = argmax P(data|θ) — find single best weights, output single prediction.
- **Bayesian Deep Learning**: P(y|x, data) = ∫ P(y|x, θ) P(θ|data) dθ — average over all plausible weight configurations weighted by posterior probability.
- **Core Challenge**: For networks with millions of parameters, computing the true posterior is computationally intractable — requiring approximation methods.
**Bayes' Rule Applied to Networks**
P(θ|data) = P(data|θ) × P(θ) / P(data)
- **Prior P(θ)**: Beliefs about weights before seeing data (typically Gaussian: weight regularization is a Gaussian prior).
- **Likelihood P(data|θ)**: How well weights explain training data (cross-entropy loss is negative log-likelihood).
- **Posterior P(θ|data)**: Updated beliefs about weights after seeing data — the target distribution.
- **Marginal Likelihood P(data)**: Normalizing constant — computationally intractable for large networks.
**Why Bayesian Deep Learning Matters**
- **Epistemic Uncertainty**: The posterior spread over weights naturally represents the model's uncertainty about what the correct weights are — wide posterior = high epistemic uncertainty = model doesn't have enough data to be confident.
- **Out-of-Distribution Detection**: When test inputs fall outside the training distribution, the posterior predictive variance is high — the model correctly expresses uncertainty on novel inputs rather than outputting overconfident wrong answers.
- **Active Learning**: Epistemic uncertainty from the posterior identifies which unlabeled examples would most reduce posterior uncertainty — directing data collection efficiently.
- **Catastrophic Forgetting**: Bayesian methods like EWC (Elastic Weight Consolidation) use the Fisher information matrix (approximation of posterior curvature) to prevent overwriting important weights during continual learning.
- **Scientific Applications**: In physics, chemistry, and biology, Bayesian neural networks provide calibrated uncertainties for surrogate models — uncertainty estimates guide which expensive experiments to run next.
**Approximation Methods**
**Variational Inference (Mean-Field)**:
- Approximate posterior P(θ|data) with a factored Gaussian Q(θ) = ∏ N(μ_i, σ_i²).
- Optimize ELBO (evidence lower bound): L = E_Q[log P(data|θ)] - KL(Q||P(θ)).
- Results in "Bayes by Backprop" (Blundell et al.) — each weight has learnable mean and variance.
- Limitation: Mean-field assumption ignores weight correlations; underestimates posterior uncertainty.
**Laplace Approximation**:
- Train network normally to find θ* (MAP estimate).
- Fit a Gaussian at θ* using the Hessian of the loss: P(θ|data) ≈ N(θ*, H⁻¹).
- Modern approach (Daxberger et al.): Last-layer Laplace is computationally feasible for large networks.
**Monte Carlo Dropout (Practical Gold Standard)**:
- Gal & Ghahramani (2016): Dropout training + dropout at inference = approximate Bayesian inference.
- Run T stochastic forward passes; mean = prediction; variance = uncertainty.
- No architecture change required — instant Bayesian uncertainty from any dropout-trained network.
**Deep Ensembles**:
- Train N networks from different random initializations.
- Lakshminarayanan et al. (2017): Ensembles are not Bayesian but empirically outperform most Bayesian approximations.
- Simple, parallelizable, and often the best practical uncertainty method.
**Bayesian Deep Learning vs. Alternatives**
| Method | Theoretical Grounding | Computational Cost | Calibration Quality |
|--------|----------------------|-------------------|---------------------|
| Bayesian NN (VI) | High | High (2x parameters) | Good |
| Laplace Approximation | High | Medium | Good |
| MC Dropout | Moderate | Low | Moderate |
| Deep Ensembles | Low | Medium (N× training) | Very Good |
| Temperature Scaling | None | Very Low | Moderate |
| Conformal Prediction | None (frequentist) | Very Low | Guaranteed |
Bayesian deep learning is **the principled framework for uncertainty-aware neural networks** — by maintaining distributions over weights rather than point estimates, Bayesian models genuinely know what they don't know, providing the epistemic foundation for trustworthy AI in scientific, medical, and safety-critical applications where confidence calibration is as important as prediction accuracy.