Bayesian Inference for Engineering Applications
# Bayesian Inference for Engineering Applications
## Introduction & Motivation
Bayesian inference provides principled methods for uncertainty quantification and parameter estimation under limited data. Critical for engineering decisions with high stakes, Bayesian methods combine prior knowledge with experimental data to make robust predictions and guide optimal decision-making.
Motivation: Apply Bayesian inference for robust engineering decision-making.
Applications: Parameter estimation, uncertainty quantification, model selection, anomaly detection.
---
## Core Concepts & Theory
### Prior Distributions
Encoding prior knowledge.
### Likelihood Functions
Data-dependent probability.
### Posterior Distribution
Updated belief after data.
### Model Comparison
Bayes factors and evidence.
---
## Mathematical Formulation
Bayes' Rule:
$$P( heta|D) = \frac{P(D| heta)P( heta)}{P(D)} = \frac{P(D| heta)P( heta)}{\int P(D| heta)P( heta)d heta}$$
Posterior Predictive:
$$P( ilde{y}|D) = \int P( ilde{y}| heta)P( heta|D)d heta$$
Credible Interval:
$$P( heta_L < heta < heta_U | D) = 1 - \alpha$$
---
## Advanced Theory & Extensions
### MCMC Sampling
Markov Chain Monte Carlo methods.
### Variational Inference
Approximate posterior distributions.
### Hierarchical Models
Multi-level Bayesian structures.
---
## Computational Considerations
MCMC: O(I·D) for I iterations, D dimensions.
Variational Inference: O(N·D²) for N data points.
Model Comparison: O(M·C) for M models, C calculations.
---
## Practical Implementation Strategies
### Prior Selection
Informative vs. non-informative priors.
### Posterior Sampling
MCMC chains and diagnostics.
### Credible Intervals
Quantile-based inference.
---
## Benchmark Datasets & Evaluation
Synthetic Datasets: Controlled validation.
Industrial Data: Real engineering measurements.
Benchmark Problems: Standard inference tests.
---
## Key Challenges & Limitations
### High Dimensionality
Posterior computation in many dimensions.
### Prior Sensitivity
Dependence on prior specification.
### Convergence Diagnostics
Assessing chain quality.
---
## Hyperparameter Tuning
MCMC chains: 1000-10000 iterations.
Burn-in period: 10-20% of iterations.
Thinning factor: 1-10.
---
## Real-World Applications & Case Studies
Materials Properties: Parameter uncertainty.
Process Control: Optimal setpoints with uncertainty.
Sensor Fusion: Multi-sensor data integration.
---
## Integration with Other Methods
Bayesian inference + ML models; + optimization; + decision theory.
---
## Summary & Key Takeaways
Bayesian inference enables robust decision-making under uncertainty.
Principles:
1. Prior: Encode domain knowledge.
2. Likelihood: Model data generating process.
3. Posterior: Update beliefs from data.
4. Inference: Compute posterior quantities.
5. Decision: Make robust choices.
---
## Appendix: Practical Labs
### Lab 1: Conjugate Prior Analysis
import numpy as np
class ConjugateBetaBinomial:
def __init__(self, alpha=1, beta=1):
# Prior: Beta(α, β)
self.alpha = alpha
self.beta = beta
def update_posterior(self, successes, trials):
"""Update posterior with binomial data"""
# Posterior: Beta(α + successes, β + failures)
failures = trials - successes
alpha_post = self.alpha + successes
beta_post = self.beta + failures
return alpha_post, beta_post
def posterior_mean(self, alpha_post, beta_post):
"""Mean of posterior distribution"""
return alpha_post / (alpha_post + beta_post)
def posterior_credible_interval(self, alpha_post, beta_post, confidence=0.95):
"""Credible interval for parameter"""
# Simplified: use normal approximation
mean = self.posterior_mean(alpha_post, beta_post)
variance = (alpha_post * beta_post) / ((alpha_post + beta_post)**2 * (alpha_post + beta_post + 1))
z = 1.96 # 95% CI
ci_lower = mean - z * np.sqrt(variance)
ci_upper = mean + z * np.sqrt(variance)
return ci_lower, ci_upper
model = ConjugateBetaBinomial(alpha=1, beta=1)
# Observe data
successes = 7
trials = 10
alpha_post, beta_post = model.update_posterior(successes, trials)
mean_post = model.posterior_mean(alpha_post, beta_post)
ci_low, ci_high = model.posterior_credible_interval(alpha_post, beta_post)
print(f"✓ Posterior after {successes}/{trials} successes")
print(f" Mean: {mean_post:.3f}")
print(f" 95% CI: [{ci_low:.3f}, {ci_high:.3f}]")### Lab 2: MCMC Sampling (Metropolis-Hastings)
import numpy as np
class MetropolisHastingsSampler:
def __init__(self, log_posterior_fn, proposal_std=1.0):
self.log_posterior = log_posterior_fn
self.proposal_std = proposal_std
def sample(self, initial_theta, n_iterations=5000):
"""Run MH sampler"""
theta = initial_theta.copy()
samples = [theta.copy()]
accepted = 0
for iteration in range(n_iterations):
# Propose new theta
theta_proposed = theta + np.random.randn(len(theta)) * self.proposal_std
# Compute acceptance ratio
log_alpha = (self.log_posterior(theta_proposed) -
self.log_posterior(theta))
# Accept/reject
if np.log(np.random.uniform()) < log_alpha:
theta = theta_proposed
accepted += 1
samples.append(theta.copy())
acceptance_rate = accepted / n_iterations
return np.array(samples), acceptance_rate
def log_posterior_normal(theta, y=0.0, sigma=1.0):
"""Log posterior for normal model"""
# Simple case: normal likelihood + normal prior
log_likelihood = -0.5 * ((y - theta[0]) / sigma) ** 2
log_prior = -0.5 * theta[0] ** 2 # Prior: N(0, 1)
return log_likelihood + log_prior
sampler = MetropolisHastingsSampler(log_posterior_normal, proposal_std=0.5)
samples, acc_rate = sampler.sample(np.array([0.0]), n_iterations=5000)
print(f"✓ MCMC sampling complete")
print(f" Acceptance rate: {acc_rate:.1%}")
print(f" Posterior mean (excluding burn-in): {np.mean(samples[1000:]):.3f}")
print(f" Posterior std: {np.std(samples[1000:]):.3f}")### Lab 3: Variational Inference
import numpy as np
class VariationalInferenceBernoulli:
def __init__(self, data, prior_alpha=1, prior_beta=1):
self.data = data
self.prior_alpha = prior_alpha
self.prior_beta = prior_beta
# Variational parameters
self.q_alpha = prior_alpha
self.q_beta = prior_beta
def elbo(self):
"""Compute evidence lower bound"""
n_success = np.sum(self.data)
n_total = len(self.data)
# Expected log likelihood
exp_log_p = n_success * (np.log(self.q_alpha) - np.log(self.q_alpha + self.q_beta))
exp_log_p += (n_total - n_success) * (np.log(self.q_beta) - np.log(self.q_alpha + self.q_beta))
# KL divergence (Beta distributions)
kl_divergence = 0 # Simplified: ignore for now
return exp_log_p
def optimize_variational(self, iterations=100, lr=0.01):
"""Optimize variational parameters"""
for _ in range(iterations):
# Compute gradients (simplified coordinate ascent)
n_success = np.sum(self.data)
n_total = len(self.data)
# Update q_alpha
self.q_alpha = self.prior_alpha + n_success
# Update q_beta
self.q_beta = self.prior_beta + (n_total - n_success)
model = VariationalInferenceBernoulli(np.random.binomial(1, 0.6, 100))
model.optimize_variational(iterations=10)
posterior_mean = model.q_alpha / (model.q_alpha + model.q_beta)
print(f"✓ Variational posterior mean: {posterior_mean:.3f}")### Lab 4: Integrated Bayesian Inference System
import numpy as np
class BayesianEngineeringSystem:
def __init__(self, n_sensors=5):
self.n_sensors = n_sensors
# Prior on true state
self.state_prior_mean = np.zeros(n_sensors)
self.state_prior_cov = np.eye(n_sensors)
# Sensor noise
self.sensor_noise_cov = 0.1 * np.eye(n_sensors)
def process_sensor_data(self, measurements, n_posterior_samples=1000):
"""Update beliefs from sensor measurements"""
# Posterior mean (simplified Kalman-like)
H = np.eye(n_sensors) # Measurement matrix
# Compute posterior covariance
S = H @ self.state_prior_cov @ H.T + self.sensor_noise_cov
K = self.state_prior_cov @ H.T @ np.linalg.inv(S) # Kalman gain
# Posterior mean
innovation = measurements - H @ self.state_prior_mean
state_posterior_mean = self.state_prior_mean + K @ innovation
# Posterior covariance
state_posterior_cov = (np.eye(self.n_sensors) - K @ H) @ self.state_prior_cov
return state_posterior_mean, state_posterior_cov
def estimate_parameter(self, measurements):
"""Estimate engineering parameter with uncertainty"""
posterior_mean, posterior_cov = self.process_sensor_data(measurements)
posterior_std = np.sqrt(np.diag(posterior_cov))
return posterior_mean, posterior_std
def compute_optimal_decision(self, posterior_mean, cost_matrix):
"""Make optimal decision under uncertainty"""
# Minimize expected cost
expected_costs = cost_matrix @ posterior_mean
optimal_action = np.argmin(expected_costs)
return optimal_action
system = BayesianEngineeringSystem(n_sensors=3)
measurements = np.random.randn(3) + 2.0
posterior_mean, posterior_std = system.estimate_parameter(measurements)
print(f"✓ Posterior estimates:")
for i, (mean, std) in enumerate(zip(posterior_mean, posterior_std)):
print(f" Sensor {i}: {mean:.2f} ± {std:.2f}")
cost_matrix = np.random.randn(4, 3)
action = system.compute_optimal_decision(posterior_mean, cost_matrix)
print(f"✓ Optimal action: {action}")---