Bayesian Nonparametric Methods

# Bayesian Nonparametric Methods

## Introduction & Motivation

Bayesian nonparametric methods provide flexible modeling where complexity adapts to data. Essential for discovering underlying structure in materials data, process systems, and scientific measurements without pre-specifying model complexity.

Motivation: Flexible Bayesian inference with data-driven complexity.

Applications: Flexible density estimation, clustering, regression, structure discovery.

---

## Core Concepts & Theory

### Infinite-Dimensional Priors

Unbounded model complexity.

### Exchangeability

De Finetti representation.

### Latent Variables

Hidden structure inference.

### Complexity Adaptation

Automatic model selection.

---

## Mathematical Formulation

Exchangeable Sequence:
$$P(X_1, \ldots, X_n) = P(X_{\sigma(1)}, \ldots, X_{\sigma(n)})$$

De Finetti:
$$X_1, X_2, \ldots | G \sim ext{i.i.d. } G, \quad G \sim ext{random measure}$$

Posterior Concentration:
$$P( heta \in N( heta_0) | X^n) o 1 ext{ as } n o \infty$$

---

## Advanced Theory & Extensions

### Beta-Bernoulli Process

Feature allocation.

### Indian Buffet Process

Sparse feature matrices.

### Gaussian Process Priors

Function-space inference.

---

## Computational Considerations

Posterior Sampling: O(N·K) MCMC.

Variational: O(N·D) per iteration.

Prediction: O(K·D) per point.

---

## Practical Implementation Strategies

### Truncation

Finite approximations.

### Sampling Methods

Collapsed Gibbs, variational.

### Posterior Inference

Mean, credible sets.

---

## Benchmark Datasets & Evaluation

Synthetic Data: Validation.

UCI Datasets: Benchmark comparison.

Domain Data: Application studies.

---

## Key Challenges & Limitations

### Computational Burden

MCMC slowness.

### Approximation Error

Truncation effects.

### Model Comparison

Marginal likelihood computation.

---

## Hyperparameter Tuning

Concentration: Problem-dependent.

Truncation level: Adaptive.

Priors: Regularization.

---

## Real-World Applications & Case Studies

Clustering: Flexible mixture models.

Regression: Nonparametric smoothing.

Classification: Infinite mixture classifiers.

---

## Integration with Other Methods

Bayesian nonparametrics + machine learning; + mixture models; + hierarchical methods.

---

## Summary & Key Takeaways

Bayesian nonparametrics enable adaptive modeling.

Principles:
1. Infinity: Unbounded complexity.
2. Exchangeability: Data symmetry.
3. Adaptation: Data-driven structure.
4. Flexibility: No fixed model.
5. Uncertainty: Full posterior.

---

## Appendix: Practical Labs

### Lab 1: Beta-Bernoulli Process

import numpy as np

def beta_bernoulli_process(alpha, n_customers, n_features=10):
 """Simulate Beta-Bernoulli process"""
 features = []
 
 for feature in range(n_features):
 prob = np.random.beta(alpha, 1)
 presence = np.random.binomial(1, prob, n_customers)
 features.append(presence)
 
 # Infinite features
 while np.random.rand() < alpha / (alpha + n_customers):
 new_feature = np.random.binomial(1, 0.5, n_customers)
 features.append(new_feature)
 
 return np.array(features).T

features = beta_bernoulli_process(alpha=1.0, n_customers=20, n_features=5)
print(f"✓ Feature matrix shape: {features.shape}")

### Lab 2: Indian Buffet Process

import numpy as np

def indian_buffet_process(alpha, n_customers):
 """Simulate IBP"""
 dishes = []
 
 for customer in range(n_customers):
 # Sample dishes
 dishes_tried = []
 
 for dish in range(len(dishes)):
 prob = dishes[dish] / (customer + 1)
 if np.random.rand() < prob:
 dishes_tried.append(dish)
 
 # New dishes
 n_new = np.random.poisson(alpha / (customer + 1))
 dishes_tried.extend(range(len(dishes), len(dishes) + n_new))
 
 # Update dish counts
 for dish in dishes_tried:
 if dish >= len(dishes):
 dishes.append(1)
 else:
 dishes[dish] += 1
 
 return np.array(dishes)

dishes = indian_buffet_process(alpha=1.0, n_customers=20)
n_features = len(dishes)

print(f"✓ IBP discovered {n_features} features")

### Lab 3: Gaussian Process Regression

import numpy as np

class GaussianProcessNP:
 def __init__(self, lengthscale=1.0, noise=0.1):
 self.ls = lengthscale
 self.noise = noise
 self.X_train = None
 self.y_train = None
 
 def fit(self, X, y):
 """Fit GP"""
 self.X_train = X
 self.y_train = y
 
 def predict(self, X_test):
 """GP prediction"""
 if self.X_train is None:
 return np.mean(self.y_train) * np.ones(len(X_test))
 
 # Kernel
 K = np.exp(-np.sum((self.X_train[:, None] - self.X_train[None, :])**2, axis=2) / (2*self.ls**2))
 Ks = np.exp(-np.sum((self.X_train[None, :] - X_test[:, None])**2, axis=2) / (2*self.ls**2))
 
 # Predictive
 L = np.linalg.cholesky(K + np.eye(len(self.X_train)) * self.noise)
 mu = Ks.T @ np.linalg.solve(L, np.linalg.solve(L.T, self.y_train))
 
 return mu

gp = GaussianProcessNP()
X = np.random.uniform(-5, 5, 20)
y = np.sin(X) + np.random.randn(20) * 0.1

gp.fit(X, y)
X_test = np.linspace(-5, 5, 50)
y_pred = gp.predict(X_test)

print(f"✓ GP predictions: {y_pred.shape}")

### Lab 4: Infinite Mixture

import numpy as np

class InfiniteMixture:
 def __init__(self, alpha=1.0):
 self.alpha = alpha
 self.components = []
 self.weights = []
 
 def add_component(self, mean, cov):
 """Add component"""
 self.components.append((mean, cov))
 self.weights.append(1.0)
 self.weights = np.array(self.weights) / np.sum(self.weights)
 
 def sample(self, n_samples=100):
 """Sample from infinite mixture"""
 samples = []
 
 for _ in range(n_samples):
 # CRP allocation
 if np.random.rand() < self.alpha / (len(self.components) + self.alpha):
 new_component_idx = len(self.components)
 self.add_component(np.random.randn(2), np.eye(2))
 else:
 new_component_idx = np.random.choice(len(self.components), p=self.weights)
 
 mean, cov = self.components[new_component_idx]
 sample = np.random.multivariate_normal(mean, cov)
 samples.append(sample)
 
 return np.array(samples)

im = InfiniteMixture(alpha=1.0)
samples = im.sample(n_samples=100)

print(f"✓ Infinite mixture: {len(im.components)} components, {samples.shape[0]} samples")

---