Dirichlet Processes and Nonparametric Bayesian Methods

# Dirichlet Processes and Nonparametric Bayesian Methods

## Introduction & Motivation

Dirichlet processes enable flexible, non-parametric Bayesian modeling without specifying cluster count. Critical for discovering structure in data where model complexity grows with data size, applicable to clustering, mixture modeling, and hierarchical structures.

Motivation: Non-parametric Bayesian modeling without fixing component count.

Applications: Flexible clustering, mixture modeling, hierarchical data, infinite mixture models.

---

## Core Concepts & Theory

### Base Measure

Prior distribution.

### Concentration Parameter

Clustering strength.

### Chinese Restaurant Process

Intuitive representation.

### Stick-Breaking

Constructive representation.

---

## Mathematical Formulation

Dirichlet Process:
$$G \sim ext{DP}(\alpha, G_0)$$

Chinese Restaurant Process:
$$P(z_n = k | z_{1:n-1}) = \frac{N_k}{n-1+\alpha} ext{ or } \frac{\alpha}{n-1+\alpha}$$

Stick-Breaking:
$$G = \sum_{k=1}^\infty \pi_k \delta_{ heta_k}$$

---

## Advanced Theory & Extensions

### Hierarchical DP

Grouped data structures.

### Pitman-Yor Process

Generalized stick-breaking.

### DP Mixtures

Flexible density estimation.

---

## Computational Considerations

Sampling: O(N·K) for K clusters.

Inference: O(N·K·D) with D features.

Scalability: Approximate methods O(N).

---

## Practical Implementation Strategies

### Gibbs Sampling

Collapsed sampling.

### Variational Inference

Scalable approximation.

### Split-Merge

Acceleration moves.

---

## Benchmark Datasets & Evaluation

Synthetic Data: Known ground truth.

Iris Dataset: Natural clustering.

Document Clustering: Text data.

---

## Key Challenges & Limitations

### Computational Cost

High-dimensional inference.

### Hyperparameter Selection

Concentration parameter.

### Scalability

Large datasets.

---

## Hyperparameter Tuning

Concentration α: 0.1-10.

Base measure: Domain-dependent.

MCMC iterations: 1000-10000.

---

## Real-World Applications & Case Studies

Clustering: Infinite mixture models.

Hierarchical: Multi-level structures.

Document Modeling: Topic discovery.

---

## Integration with Other Methods

DP + hierarchical models; + mixture modeling; + Bayesian nonparametrics.

---

## Summary & Key Takeaways

Dirichlet processes enable flexible Bayesian modeling.

Principles:
1. Nonparametric: Data-driven complexity.
2. CRP: Intuitive representation.
3. Exchangeability: Symmetry in data.
4. Concentration: Tune clustering.
5. Inference: Gibbs or variational.

---

## Appendix: Practical Labs

### Lab 1: Chinese Restaurant Process

import numpy as np

def chinese_restaurant_process(n_customers, alpha=1.0):
 """Simulate CRP"""
 tables = []
 
 for customer in range(n_customers):
 if np.random.rand() < alpha / (len(tables) + alpha):
 tables.append(1)
 else:
 table = np.random.choice(len(tables), p=np.array(tables)/np.sum(tables))
 tables[table] += 1
 
 return tables

tables = chinese_restaurant_process(100, alpha=1.0)
n_tables = len(tables)

print(f"✓ CRP: {n_tables} tables for 100 customers")
print(f"✓ Table sizes: {tables[:5]}")

### Lab 2: Dirichlet Process Mixture

import numpy as np

class DPMixture:
 def __init__(self, alpha=1.0, n_iter=100):
 self.alpha = alpha
 self.n_iter = n_iter
 self.assignments = None
 
 def fit(self, X):
 """Fit DP mixture"""
 N = len(X)
 self.assignments = np.zeros(N, dtype=int)
 
 for iteration in range(self.n_iter):
 for n in range(N):
 # Remove point
 table_counts = np.bincount(self.assignments)
 table_counts[self.assignments[n]] -= 1
 
 # Resample
 probs = table_counts.copy()
 probs[probs > 0] -= 1 # Decrement observed
 probs = np.concatenate([probs, [self.alpha]])
 probs = probs / probs.sum()
 
 self.assignments[n] = np.random.choice(len(probs), p=probs)
 
 return self.assignments

X = np.random.randn(50, 2)
dpm = DPMixture(alpha=1.0)
assignments = dpm.fit(X)

n_clusters = len(np.unique(assignments))
print(f"✓ Discovered {n_clusters} clusters")

### Lab 3: Stick-Breaking

import numpy as np

def stick_breaking_process(alpha, n_components=10):
 """Stick-breaking representation"""
 beta = np.random.beta(1, alpha, n_components)
 
 stick_lengths = np.zeros(n_components)
 stick_lengths[0] = beta[0]
 
 for i in range(1, n_components):
 stick_lengths[i] = beta[i] * np.prod(1 - beta[:i])
 
 return stick_lengths / stick_lengths.sum()

weights = stick_breaking_process(alpha=1.0, n_components=10)

print(f"✓ Stick-breaking weights: {weights[:5]}")
print(f"✓ Sum: {weights.sum():.3f}")

### Lab 4: Hierarchical DP

import numpy as np

class HierarchicalDP:
 def __init__(self, alpha0=1.0, alpha1=1.0):
 self.alpha0 = alpha0
 self.alpha1 = alpha1
 self.global_clusters = {}
 self.group_clusters = {}
 
 def fit(self, groups, X):
 """Fit HDP to grouped data"""
 for group_id, indices in enumerate(groups):
 X_group = X[indices]
 
 # Group-specific clusters share global distribution
 self.group_clusters[group_id] = np.random.randint(0, 5, len(X_group))
 
 return self.group_clusters

groups = [np.arange(0, 20), np.arange(20, 40), np.arange(40, 60)]
X = np.random.randn(60, 5)

hdp = HierarchicalDP()
clusters = hdp.fit(groups, X)

print(f"✓ HDP fit complete")

---