Bayesian Neural Networks Uncertainty Probabilistic Deep Learning
# Bayesian Neural Networks: Uncertainty & Probabilistic Deep Learning
## Introduction & Motivation
Bayesian Neural Networks: place prior distributions over weights. Posterior inference; uncertainty quantification. Variational inference approximates posterior. Applications: uncertainty estimation, out-of-distribution detection, active learning.
Motivation: Capture model uncertainty; avoid overconfidence.
Applications: Uncertainty, robustness, calibration.
---
## Core Concepts & Theory
### Weight Priors
Distribution over network weights.
### Posterior Distribution
Updated weight distribution given data.
### Variational Inference
Approximate posterior with tractable distribution.
---
## Mathematical Formulation
Bayesian neural network:
$$P(w|D) = \frac{P(D|w)P(w)}{P(D)}$$
ELBO (Evidence Lower Bound):
$$\mathcal{L} = \mathbb{E}_{q(w)}[\log P(D|w)] - ext{KL}(q(w) \| p(w))$$
Prediction with uncertainty:
$$P(y|x, D) = \int P(y|x, w) P(w|D) dw$$
---
## Advanced Theory & Extensions
### Variational Dropout
Dropout as approximate Bayesian inference.
### SWAG (Stochastic Weight Averaging-Gaussian)
Posterior approximation from SGD trajectory.
### Laplace Approximation
Posterior Gaussian around MAP estimate.
---
## Computational Considerations
ELBO computation: O(batch_size·model_size·samples).
MC sampling: O(num_samples) for prediction.
Posterior approximation: O(model_size²) for Laplace.
---
## Practical Implementation Strategies
### Variational Posterior Family
Diagonal Gaussian; mean-field assumption.
### Prior Selection
Standard normal; promotes regularization.
### Reparameterization Trick
Backprop through sampling.
---
## Benchmark Datasets & Evaluation
UCI Datasets: Uncertainty benchmarks.
MNIST, CIFAR-10: Classification with uncertainty.
OOD Detection: Robustness evaluation.
---
## Key Challenges & Limitations
### Posterior Approximation
Mean-field assumption oversimplifies.
### Computational Cost
Multiple forward passes needed.
### Hyperparameter Selection
Prior variance choice impacts learning.
---
## Hyperparameter Tuning
Prior variance: 1.0-10.0; regularization strength.
KL weight (annealing): 0-1; balance fit and prior.
Number of samples: 10-50; uncertainty estimate quality.
---
## Real-World Applications & Case Studies
Uncertainty Estimation: Confidence in predictions.
Active Learning: Query informative samples.
OOD Detection: Identify unfamiliar inputs.
---
## Integration with Other Methods
BNN + Ensemble → improved uncertainty.
BNN + Transfer → probabilistic adaptation.
---
## Summary & Key Takeaways
Bayesian Neural Networks via probabilistic weight distributions enable uncertainty quantification through variational inference and posterior approximation.
Principles:
1. Weight priors: Bayesian framework.
2. Posterior: data-dependent distribution.
3. ELBO: tractable objective.
4. KL divergence: prior compliance.
5. MC sampling: prediction uncertainty.
---
---
## Appendix: Practical Labs
### Lab 1: Reparameterization Trick
import numpy as np
def reparameterized_sampling(mu, log_sigma, num_samples=10):
"""Reparameterization trick for Gaussian posterior"""
sigma = np.exp(log_sigma)
samples = []
for _ in range(num_samples):
epsilon = np.random.randn(*mu.shape)
w = mu + sigma * epsilon
samples.append(w)
return np.array(samples)
# Test
np.random.seed(42)
mu = np.random.randn(10)
log_sigma = np.random.randn(10)
samples = reparameterized_sampling(mu, log_sigma, 100)
assert samples.shape == (100, 10), "Samples shape"
assert np.isfinite(samples).all(), "Samples finite"
print("✓ Reparameterization working")
if __name__ == "__main__":
print("Lab 1: Reparameterization - PASSED")### Lab 2: KL Divergence (Gaussian)
import numpy as np
def kl_divergence_gaussian(mu_q, log_sigma_q, mu_p, log_sigma_p):
"""KL divergence between two Gaussians"""
sigma_q = np.exp(log_sigma_q)
sigma_p = np.exp(log_sigma_p)
# KL divergence components
kl = 0.5 * np.sum(
log_sigma_p - log_sigma_q
+ (sigma_q**2 + (mu_q - mu_p)**2) / (sigma_p**2)
- 1
)
return kl
# Test
np.random.seed(42)
mu_q = np.random.randn(10)
log_sigma_q = np.random.randn(10)
mu_p = np.zeros(10)
log_sigma_p = np.zeros(10)
kl = kl_divergence_gaussian(mu_q, log_sigma_q, mu_p, log_sigma_p)
assert kl >= 0, "KL divergence non-negative"
assert np.isfinite(kl), "KL divergence finite"
print("✓ KL divergence working")
if __name__ == "__main__":
print("Lab 2: KLDivergence - PASSED")### Lab 3: ELBO Computation
import numpy as np
def compute_elbo(weights, logits, targets, mu, log_sigma, mu_p=None, log_sigma_p=None):
"""Compute ELBO for Bayesian NN"""
if mu_p is None:
mu_p = np.zeros_like(mu)
if log_sigma_p is None:
log_sigma_p = np.zeros_like(log_sigma)
# Likelihood term
probs = 1 / (1 + np.exp(-logits))
likelihood = np.sum(targets * np.log(probs + 1e-8) + (1-targets) * np.log(1-probs + 1e-8))
# KL term
sigma_q = np.exp(log_sigma)
sigma_p = np.exp(log_sigma_p)
kl = 0.5 * np.sum(
log_sigma_p - log_sigma
+ (sigma_q**2 + (mu - mu_p)**2) / (sigma_p**2)
- 1
)
# ELBO
elbo = likelihood - kl
return elbo
# Test
np.random.seed(42)
weights = np.random.randn(10)
logits = np.random.randn(32)
targets = np.random.randint(0, 2, 32).astype(float)
mu = np.random.randn(10)
log_sigma = np.random.randn(10)
elbo = compute_elbo(weights, logits, targets, mu, log_sigma)
assert np.isfinite(elbo), "ELBO finite"
print("✓ ELBO computation working")
if __name__ == "__main__":
print("Lab 3: ELBOComputation - PASSED")### Lab 4: Uncertainty Estimation
import numpy as np
def predict_with_uncertainty(forward_fn, x, num_samples=100):
"""Make predictions with uncertainty quantification"""
predictions = []
for _ in range(num_samples):
# Forward pass with stochastic weights
pred = forward_fn(x)
predictions.append(pred)
predictions = np.array(predictions)
# Mean and variance
mean = predictions.mean(axis=0)
std = predictions.std(axis=0)
return mean, std
# Test
np.random.seed(42)
def dummy_forward(x):
return np.random.randn(len(x)) + 0.5
x = np.random.randn(32)
mean, std = predict_with_uncertainty(dummy_forward, x, 50)
assert mean.shape == (32,), "Mean shape"
assert std.shape == (32,), "Std shape"
assert np.all(std >= 0), "Std non-negative"
print("✓ Uncertainty estimation working")
if __name__ == "__main__":
print("Lab 4: UncertaintyEstimation - PASSED")