Bayesian Methods Probabilistic Inference Prior-Posterior
# Bayesian Methods: Probabilistic Inference & Prior-Posterior
## Introduction & Motivation
Bayesian methods combine prior beliefs with observed data via Bayes theorem: P(θ|D) ∝ P(D|θ) P(θ). Posterior updates beliefs as evidence accumulates. Variational inference, MCMC approximate intractable posteriors. Enables uncertainty quantification, principled model comparison (marginal likelihood).
Motivation: Frequentist methods lack uncertainty; Bayesian naturally quantifies posterior belief and model plausibility. Enables decision-making under uncertainty.
Applications: Medical diagnosis, recommendation systems, A/B testing, anomaly detection.
---
## Core Concepts & Theory
### Bayes Theorem
P(θ|D) = P(D|θ)P(θ) / P(D).
Posterior ∝ Likelihood × Prior.
### Marginal Likelihood (Evidence)
P(D) = ∫ P(D|θ) P(θ) dθ.
Normalizer; useful for model comparison.
### Variational Inference
Approximate posterior q(θ) via ELBO: log P(D) ≥ E_q[log P(D|θ)] - KL(q||p).
---
## Mathematical Formulation
Bayes theorem:
$$P( heta | D) = \frac{P(D | heta) P( heta)}{P(D)} = \frac{P(D | heta) P( heta)}{\int P(D | heta) P( heta) d heta}$$
Log posterior (unnormalized):
$$\log P( heta | D) \propto \sum_{i=1}^n \log P(y_i | x_i, heta) + \log P( heta)$$
Variational lower bound (ELBO):
$$\mathcal{L}(q) = \mathbb{E}_q[\log P(D, heta)] - \mathbb{E}_q[\log q( heta)] = \mathbb{E}_q[\log P(D| heta)] - ext{KL}(q( heta) \| p( heta))$$
---
## Advanced Theory & Extensions
### Hierarchical Bayesian Models
Hyperpriors on prior parameters: P(α) → P(θ|α) → P(D|θ).
Automatic regularization strength tuning.
### Empirical Bayes
Maximize marginal likelihood w.r.t. hyperparameters: α* = argmax_α P(D|α).
### Approximate Bayesian Computation (ABC)
When likelihood intractable; accept samples if simulated ≈ observed.
---
## Computational Considerations
MCMC: O(iterations × d) per draw; Metropolis-Hastings, Gibbs, HMC.
Variational inference: One-pass optimization; faster but biased.
Laplace approximation: O(d³) for Hessian; posterior ≈ Gaussian.
---
## Practical Implementation Strategies
### Prior Selection
Conjugate priors enable closed-form posteriors (Beta-Binomial, Normal-Normal).
Weakly informative priors: minimize assumption while stabilizing inference.
### MCMC Diagnostics
R-hat < 1.05 (convergence); effective sample size > 400.
Trace plots: no trends, good mixing.
### Posterior Visualization
Marginal distributions, credible intervals, pairwise scatter plots.
---
## Benchmark Datasets & Evaluation
Synthetic data: Known posteriors for validation.
Real datasets: Compare predictive accuracy, calibration, uncertainty estimates.
Metrics: Log predictive density, RMSE + uncertainty, coverage of credible intervals.
---
## Key Challenges & Limitations
### Computational Cost
MCMC mixes slowly in high dimensions. Hamiltonian MCMC helps.
### Prior Sensitivity
Results sensitive to prior; weak priors may underestimate uncertainty.
### Posterior Multimodality
Multiple modes; chain may get stuck in one.
---
## Hyperparameter Tuning
Prior variance: Balance informativeness vs. flexibility.
MCMC warmup: 50% of iterations discarded.
Step size (HMC): ~0.01-0.1; inversely related to condition number.
---
## Real-World Applications & Case Studies
Clinical Trials: Bayesian hierarchical models for safety monitoring.
Recommendation: Bayesian matrix factorization; uncertainty-driven exploration.
NLP: Latent Dirichlet Allocation (LDA) for topic modeling.
---
## Integration with Other Methods
Bayesian Deep Learning: Bayesian neural networks via variational inference.
Bayesian Optimization: Model uncertainty via GP; acquire high-EI points.
---
## Summary & Key Takeaways
Bayesian methods quantify posterior belief via Bayes theorem, enabling principled inference and uncertainty quantification via MCMC or variational inference.
Principles:
1. Posterior ∝ Likelihood × Prior.
2. Marginal likelihood enables model comparison.
3. Variational inference approximates intractable posteriors.
4. MCMC provides asymptotically exact samples.
5. Credible intervals reflect posterior uncertainty.
---
---
## Appendix: Practical Labs
### Lab 1: Conjugate Prior (Beta-Binomial)
import numpy as np
from scipy.stats import binom, beta
# Observe coin flips
heads, tails = 5, 3
# Prior: Beta(α=1, β=1) (uniform)
alpha_prior, beta_prior = 1, 1
# Posterior: Beta(α + heads, β + tails)
alpha_post = alpha_prior + heads
beta_post = beta_prior + tails
# Credible interval
ci_low, ci_high = beta.ppf(0.025, alpha_post, beta_post), beta.ppf(0.975, alpha_post, beta_post)
mean_post = alpha_post / (alpha_post + beta_post)
print(f"Posterior mean: {mean_post:.4f}, 95% CI: [{ci_low:.4f}, {ci_high:.4f}]")
assert 0 <= ci_low <= mean_post <= ci_high <= 1, "CI should bracket posterior mean"
assert alpha_post > alpha_prior, "Posterior alpha should increase with heads"
print("✓ Beta-Binomial conjugate prior working")
if __name__ == "__main__":
print("Lab 1: Conjugate Prior - PASSED")### Lab 2: Maximum A Posteriori (MAP)
import numpy as np
from scipy.optimize import minimize
def log_posterior(theta, X, y, alpha=1.0):
# Likelihood: Gaussian regression
residuals = y - X @ theta
likelihood = -0.5 * np.sum(residuals**2)
# Prior: L2 regularization (Gaussian prior)
prior = -0.5 * alpha * np.sum(theta**2)
return -(likelihood + prior)
# Data
np.random.seed(42)
X = np.random.randn(30, 5)
y = np.random.randn(30)
# MAP estimate
theta_init = np.zeros(5)
result = minimize(log_posterior, theta_init, args=(X, y, 1.0))
theta_map = result.x
print(f"MAP estimate: {theta_map}")
assert len(theta_map) == 5, "Should estimate 5 parameters"
assert np.isfinite(theta_map).all(), "Should be finite"
print("✓ MAP estimation working")
if __name__ == "__main__":
print("Lab 2: MAP - PASSED")### Lab 3: Variational Inference
import numpy as np
from scipy.stats import norm
def elbo(X, y, mu, log_sigma, alpha=1.0):
# Likelihood term
sigma = np.exp(log_sigma)
residuals = y - X @ mu
likelihood = -0.5 * np.sum(residuals**2) - 0.5 * np.sum((X @ np.diag(sigma**2)) @ X.T)
# KL term (prior: standard normal)
kl = -0.5 * np.sum(1 + 2*log_sigma - mu**2 - np.exp(2*log_sigma))
return likelihood - kl
# Data
np.random.seed(42)
X = np.random.randn(20, 3)
y = np.random.randn(20)
# Variational parameters
mu = np.zeros(3)
log_sigma = np.zeros(3)
elbo_val = elbo(X, y, mu, log_sigma)
print(f"ELBO: {elbo_val:.2f}")
assert np.isfinite(elbo_val), "ELBO should be finite"
print("✓ Variational inference working")
if __name__ == "__main__":
print("Lab 3: VI - PASSED")### Lab 4: Posterior Predictive
import numpy as np
from scipy.stats import norm
def posterior_predictive(X_new, mu, sigma, noise_sigma=0.1):
# E[y_new|X_new, D]
pred_mean = X_new @ mu
# Var[y_new|X_new, D]
pred_var = np.sum((X_new * sigma)**2, axis=1) + noise_sigma**2
pred_std = np.sqrt(pred_var)
return pred_mean, pred_std
# Posterior from training
mu = np.array([1.0, -0.5])
sigma = np.array([0.2, 0.1])
# New data
X_new = np.array([[1, 0], [0, 1], [1, 1]])
pred_mean, pred_std = posterior_predictive(X_new, mu, sigma)
print(f"Predictions: mean={pred_mean}, std={pred_std}")
assert pred_mean.shape == (3,), "Should have 3 predictions"
assert (pred_std > 0).all(), "Std should be positive"
print("✓ Posterior predictive working")
if __name__ == "__main__":
print("Lab 4: Predictive - PASSED")