Bayesian Inference for Engineering Applications

# Bayesian Inference for Engineering Applications

## Introduction & Motivation

Bayesian inference provides principled methods for uncertainty quantification and parameter estimation under limited data. Critical for engineering decisions with high stakes, Bayesian methods combine prior knowledge with experimental data to make robust predictions and guide optimal decision-making.

Motivation: Apply Bayesian inference for robust engineering decision-making.

Applications: Parameter estimation, uncertainty quantification, model selection, anomaly detection.

---

## Core Concepts & Theory

### Prior Distributions

Encoding prior knowledge.

### Likelihood Functions

Data-dependent probability.

### Posterior Distribution

Updated belief after data.

### Model Comparison

Bayes factors and evidence.

---

## Mathematical Formulation

Bayes' Rule:
$$P( heta|D) = \frac{P(D| heta)P( heta)}{P(D)} = \frac{P(D| heta)P( heta)}{\int P(D| heta)P( heta)d heta}$$

Posterior Predictive:
$$P( ilde{y}|D) = \int P( ilde{y}| heta)P( heta|D)d heta$$

Credible Interval:
$$P( heta_L < heta < heta_U | D) = 1 - \alpha$$

---

## Advanced Theory & Extensions

### MCMC Sampling

Markov Chain Monte Carlo methods.

### Variational Inference

Approximate posterior distributions.

### Hierarchical Models

Multi-level Bayesian structures.

---

## Computational Considerations

MCMC: O(I·D) for I iterations, D dimensions.

Variational Inference: O(N·D²) for N data points.

Model Comparison: O(M·C) for M models, C calculations.

---

## Practical Implementation Strategies

### Prior Selection

Informative vs. non-informative priors.

### Posterior Sampling

MCMC chains and diagnostics.

### Credible Intervals

Quantile-based inference.

---

## Benchmark Datasets & Evaluation

Synthetic Datasets: Controlled validation.

Industrial Data: Real engineering measurements.

Benchmark Problems: Standard inference tests.

---

## Key Challenges & Limitations

### High Dimensionality

Posterior computation in many dimensions.

### Prior Sensitivity

Dependence on prior specification.

### Convergence Diagnostics

Assessing chain quality.

---

## Hyperparameter Tuning

MCMC chains: 1000-10000 iterations.

Burn-in period: 10-20% of iterations.

Thinning factor: 1-10.

---

## Real-World Applications & Case Studies

Materials Properties: Parameter uncertainty.

Process Control: Optimal setpoints with uncertainty.

Sensor Fusion: Multi-sensor data integration.

---

## Integration with Other Methods

Bayesian inference + ML models; + optimization; + decision theory.

---

## Summary & Key Takeaways

Bayesian inference enables robust decision-making under uncertainty.

Principles:
1. Prior: Encode domain knowledge.
2. Likelihood: Model data generating process.
3. Posterior: Update beliefs from data.
4. Inference: Compute posterior quantities.
5. Decision: Make robust choices.

---

## Appendix: Practical Labs

### Lab 1: Conjugate Prior Analysis

import numpy as np

class ConjugateBetaBinomial:
 def __init__(self, alpha=1, beta=1):
 # Prior: Beta(α, β)
 self.alpha = alpha
 self.beta = beta
 
 def update_posterior(self, successes, trials):
 """Update posterior with binomial data"""
 # Posterior: Beta(α + successes, β + failures)
 failures = trials - successes
 
 alpha_post = self.alpha + successes
 beta_post = self.beta + failures
 
 return alpha_post, beta_post
 
 def posterior_mean(self, alpha_post, beta_post):
 """Mean of posterior distribution"""
 return alpha_post / (alpha_post + beta_post)
 
 def posterior_credible_interval(self, alpha_post, beta_post, confidence=0.95):
 """Credible interval for parameter"""
 # Simplified: use normal approximation
 mean = self.posterior_mean(alpha_post, beta_post)
 variance = (alpha_post * beta_post) / ((alpha_post + beta_post)**2 * (alpha_post + beta_post + 1))
 
 z = 1.96 # 95% CI
 ci_lower = mean - z * np.sqrt(variance)
 ci_upper = mean + z * np.sqrt(variance)
 
 return ci_lower, ci_upper

model = ConjugateBetaBinomial(alpha=1, beta=1)

# Observe data
successes = 7
trials = 10

alpha_post, beta_post = model.update_posterior(successes, trials)
mean_post = model.posterior_mean(alpha_post, beta_post)
ci_low, ci_high = model.posterior_credible_interval(alpha_post, beta_post)

print(f"✓ Posterior after {successes}/{trials} successes")
print(f" Mean: {mean_post:.3f}")
print(f" 95% CI: [{ci_low:.3f}, {ci_high:.3f}]")

### Lab 2: MCMC Sampling (Metropolis-Hastings)

import numpy as np

class MetropolisHastingsSampler:
 def __init__(self, log_posterior_fn, proposal_std=1.0):
 self.log_posterior = log_posterior_fn
 self.proposal_std = proposal_std
 
 def sample(self, initial_theta, n_iterations=5000):
 """Run MH sampler"""
 theta = initial_theta.copy()
 samples = [theta.copy()]
 accepted = 0
 
 for iteration in range(n_iterations):
 # Propose new theta
 theta_proposed = theta + np.random.randn(len(theta)) * self.proposal_std
 
 # Compute acceptance ratio
 log_alpha = (self.log_posterior(theta_proposed) - 
 self.log_posterior(theta))
 
 # Accept/reject
 if np.log(np.random.uniform()) < log_alpha:
 theta = theta_proposed
 accepted += 1
 
 samples.append(theta.copy())
 
 acceptance_rate = accepted / n_iterations
 
 return np.array(samples), acceptance_rate

def log_posterior_normal(theta, y=0.0, sigma=1.0):
 """Log posterior for normal model"""
 # Simple case: normal likelihood + normal prior
 log_likelihood = -0.5 * ((y - theta[0]) / sigma) ** 2
 log_prior = -0.5 * theta[0] ** 2 # Prior: N(0, 1)
 
 return log_likelihood + log_prior

sampler = MetropolisHastingsSampler(log_posterior_normal, proposal_std=0.5)
samples, acc_rate = sampler.sample(np.array([0.0]), n_iterations=5000)

print(f"✓ MCMC sampling complete")
print(f" Acceptance rate: {acc_rate:.1%}")
print(f" Posterior mean (excluding burn-in): {np.mean(samples[1000:]):.3f}")
print(f" Posterior std: {np.std(samples[1000:]):.3f}")

### Lab 3: Variational Inference

import numpy as np

class VariationalInferenceBernoulli:
 def __init__(self, data, prior_alpha=1, prior_beta=1):
 self.data = data
 self.prior_alpha = prior_alpha
 self.prior_beta = prior_beta
 
 # Variational parameters
 self.q_alpha = prior_alpha
 self.q_beta = prior_beta
 
 def elbo(self):
 """Compute evidence lower bound"""
 n_success = np.sum(self.data)
 n_total = len(self.data)
 
 # Expected log likelihood
 exp_log_p = n_success * (np.log(self.q_alpha) - np.log(self.q_alpha + self.q_beta))
 exp_log_p += (n_total - n_success) * (np.log(self.q_beta) - np.log(self.q_alpha + self.q_beta))
 
 # KL divergence (Beta distributions)
 kl_divergence = 0 # Simplified: ignore for now
 
 return exp_log_p
 
 def optimize_variational(self, iterations=100, lr=0.01):
 """Optimize variational parameters"""
 for _ in range(iterations):
 # Compute gradients (simplified coordinate ascent)
 n_success = np.sum(self.data)
 n_total = len(self.data)
 
 # Update q_alpha
 self.q_alpha = self.prior_alpha + n_success
 
 # Update q_beta
 self.q_beta = self.prior_beta + (n_total - n_success)

model = VariationalInferenceBernoulli(np.random.binomial(1, 0.6, 100))
model.optimize_variational(iterations=10)

posterior_mean = model.q_alpha / (model.q_alpha + model.q_beta)
print(f"✓ Variational posterior mean: {posterior_mean:.3f}")

### Lab 4: Integrated Bayesian Inference System

import numpy as np

class BayesianEngineeringSystem:
 def __init__(self, n_sensors=5):
 self.n_sensors = n_sensors
 
 # Prior on true state
 self.state_prior_mean = np.zeros(n_sensors)
 self.state_prior_cov = np.eye(n_sensors)
 
 # Sensor noise
 self.sensor_noise_cov = 0.1 * np.eye(n_sensors)
 
 def process_sensor_data(self, measurements, n_posterior_samples=1000):
 """Update beliefs from sensor measurements"""
 
 # Posterior mean (simplified Kalman-like)
 H = np.eye(n_sensors) # Measurement matrix
 
 # Compute posterior covariance
 S = H @ self.state_prior_cov @ H.T + self.sensor_noise_cov
 K = self.state_prior_cov @ H.T @ np.linalg.inv(S) # Kalman gain
 
 # Posterior mean
 innovation = measurements - H @ self.state_prior_mean
 state_posterior_mean = self.state_prior_mean + K @ innovation
 
 # Posterior covariance
 state_posterior_cov = (np.eye(self.n_sensors) - K @ H) @ self.state_prior_cov
 
 return state_posterior_mean, state_posterior_cov
 
 def estimate_parameter(self, measurements):
 """Estimate engineering parameter with uncertainty"""
 posterior_mean, posterior_cov = self.process_sensor_data(measurements)
 posterior_std = np.sqrt(np.diag(posterior_cov))
 
 return posterior_mean, posterior_std
 
 def compute_optimal_decision(self, posterior_mean, cost_matrix):
 """Make optimal decision under uncertainty"""
 # Minimize expected cost
 expected_costs = cost_matrix @ posterior_mean
 optimal_action = np.argmin(expected_costs)
 
 return optimal_action

system = BayesianEngineeringSystem(n_sensors=3)

measurements = np.random.randn(3) + 2.0
posterior_mean, posterior_std = system.estimate_parameter(measurements)

print(f"✓ Posterior estimates:")
for i, (mean, std) in enumerate(zip(posterior_mean, posterior_std)):
 print(f" Sensor {i}: {mean:.2f} ± {std:.2f}")

cost_matrix = np.random.randn(4, 3)
action = system.compute_optimal_decision(posterior_mean, cost_matrix)
print(f"✓ Optimal action: {action}")

---