Mixture Models and Clustering for Data Analysis

# Mixture Models and Clustering for Data Analysis

## Introduction & Motivation

Mixture models probabilistically cluster data, assigning samples to latent components. Essential for materials characterization, process identification, customer segmentation, and unsupervised discovery in scientific and engineering applications.

Motivation: Use mixture models for principled clustering and component discovery.

Applications: Clustering, density estimation, component identification, anomaly detection.

---

## Core Concepts & Theory

### Mixture Components

Latent data clusters.

### Soft Assignment

Probabilistic membership.

### EM Algorithm

Parameter estimation.

### Model Selection

Choosing component count.

---

## Mathematical Formulation

Gaussian Mixture:
$$p(x) = \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k, \Sigma_k)$$

Posterior Responsibility:
$$\gamma(z_k|x) = \frac{\pi_k \mathcal{N}(x|\mu_k, \Sigma_k)}{\sum_j \pi_j \mathcal{N}(x|\mu_j, \Sigma_j)}$$

Log-Likelihood:
$$\ln p(X| heta) = \sum_n \ln \left\{ \sum_k \pi_k \mathcal{N}(x_n|\mu_k, \Sigma_k) ight\}$$

---

## Advanced Theory & Extensions

### Mixture of Experts

Gating networks for components.

### Hierarchical Clustering

Tree-structured mixtures.

### Variational Bayes

Approximate posteriors.

---

## Computational Considerations

E-Step: O(N·K·D).

M-Step: O(N·K·D²).

Convergence: O(I·N·K·D²) for I iterations.

---

## Practical Implementation Strategies

### Initialization

K-means or random starts.

### Convergence Criteria

Log-likelihood threshold.

### Regularization

Preventing singular covariances.

---

## Benchmark Datasets & Evaluation

Iris Dataset: Multivariate clustering.

MNIST Handwritten Digits: Image clustering.

Materials Data: Property clustering.

---

## Key Challenges & Limitations

### Model Selection

Determining K.

### Local Optima

EM convergence issues.

### Degeneracy

Singular covariances.

---

## Hyperparameter Tuning

Components: 2-10.

Covariance type: Spherical, diagonal, full.

Regularization: 1e-6 to 1e-3.

---

## Real-World Applications & Case Studies

Materials: Phase identification.

Sensors: Fault mode clustering.

Chemistry: Reaction outcome classification.

---

## Integration with Other Methods

Mixture models + Bayesian inference; + clustering; + density estimation.

---

## Summary & Key Takeaways

Mixture models enable probabilistic clustering.

Principles:
1. Model: Define mixture components.
2. Assignment: Compute posteriors.
3. Training: EM optimization.
4. Selection: Choose component count.
5. Application: Solve clustering tasks.

---

## Appendix: Practical Labs

### Lab 1: Gaussian Mixture Model

import numpy as np

class GaussianMixture:
 def __init__(self, n_components=3):
 self.n_components = n_components
 self.means = None
 self.covariances = None
 self.weights = None
 
 def initialize(self, X):
 """Initialize parameters"""
 n_samples, n_features = X.shape
 
 # Random initialization
 indices = np.random.choice(n_samples, self.n_components, replace=False)
 self.means = X[indices].copy()
 self.covariances = [np.eye(n_features) for _ in range(self.n_components)]
 self.weights = np.ones(self.n_components) / self.n_components
 
 def e_step(self, X):
 """Compute responsibilities"""
 n_samples = len(X)
 responsibilities = np.zeros((n_samples, self.n_components))
 
 for k in range(self.n_components):
 diff = X - self.means[k]
 cov = self.covariances[k] + np.eye(X.shape[1]) * 1e-6
 
 det = np.linalg.det(cov)
 inv = np.linalg.inv(cov)
 
 mahal = np.sum(diff @ inv * diff, axis=1)
 responsibilities[:, k] = self.weights[k] * np.exp(-0.5 * mahal) / np.sqrt(det)
 
 responsibilities /= (responsibilities.sum(axis=1, keepdims=True) + 1e-10)
 return responsibilities
 
 def m_step(self, X, responsibilities):
 """Update parameters"""
 n_samples = len(X)
 
 for k in range(self.n_components):
 Nk = responsibilities[:, k].sum()
 self.weights[k] = Nk / n_samples
 self.means[k] = (X * responsibilities[:, k, None]).sum(axis=0) / (Nk + 1e-10)
 
 diff = X - self.means[k]
 self.covariances[k] = (diff.T * responsibilities[:, k]) @ diff / (Nk + 1e-10)

# Test
gmm = GaussianMixture(n_components=3)
X = np.vstack([np.random.randn(30, 2), np.random.randn(30, 2) + 3])

gmm.initialize(X)
resp = gmm.e_step(X)
gmm.m_step(X, resp)

print(f"✓ Gaussian mixture model:")
print(f" Weights: {gmm.weights}")

### Lab 2: EM Algorithm

import numpy as np

def em_clustering(X, n_components=3, n_iter=100):
 """EM algorithm"""
 n_samples, n_features = X.shape
 
 # Initialize
 means = X[np.random.choice(n_samples, n_components, replace=False)]
 weights = np.ones(n_components) / n_components
 covariances = [np.eye(n_features) for _ in range(n_components)]
 
 for iteration in range(n_iter):
 # E-step
 responsibilities = np.zeros((n_samples, n_components))
 
 for k in range(n_components):
 diff = X - means[k]
 cov = covariances[k] + np.eye(n_features) * 1e-6
 inv_cov = np.linalg.inv(cov)
 
 det_cov = np.linalg.det(cov)
 mahal = np.sum(diff @ inv_cov * diff, axis=1)
 
 responsibilities[:, k] = weights[k] * np.exp(-0.5 * mahal) / np.sqrt(det_cov + 1e-10)
 
 responsibilities /= responsibilities.sum(axis=1, keepdims=True) + 1e-10
 
 # M-step
 for k in range(n_components):
 Nk = responsibilities[:, k].sum()
 weights[k] = Nk / n_samples
 means[k] = (X * responsibilities[:, k, None]).sum(axis=0) / (Nk + 1e-10)
 
 return means, weights, responsibilities

X = np.random.randn(100, 2)
means, weights, resp = em_clustering(X, n_components=2, n_iter=50)

print(f"✓ EM clustering convergence achieved")

### Lab 3: Model Selection

import numpy as np

def bic_criterion(X, n_components, ll):
 """Bayesian Information Criterion"""
 n_samples, n_features = X.shape
 n_params = n_components * (2 * n_features + 1) - 1
 
 bic = -2 * ll + n_params * np.log(n_samples)
 return bic

def aic_criterion(X, n_components, ll):
 """Akaike Information Criterion"""
 n_features = X.shape[1]
 n_params = n_components * (2 * n_features + 1) - 1
 
 aic = -2 * ll + 2 * n_params
 return aic

print(f"✓ Model selection criteria configured")

### Lab 4: Clustering Application

import numpy as np

class MaterialsClusterer:
 def __init__(self, n_materials=3):
 self.n_components = n_materials
 self.means = None
 self.labels = None
 
 def fit(self, material_properties):
 """Cluster materials"""
 n_samples, n_features = material_properties.shape
 
 # Initialize means
 self.means = material_properties[np.random.choice(n_samples, self.n_components, replace=False)]
 
 # Simple k-means-like clustering
 for _ in range(10):
 distances = np.zeros((n_samples, self.n_components))
 
 for k in range(self.n_components):
 distances[:, k] = np.linalg.norm(material_properties - self.means[k], axis=1)
 
 self.labels = np.argmin(distances, axis=1)
 
 # Update means
 for k in range(self.n_components):
 mask = self.labels == k
 if mask.sum() > 0:
 self.means[k] = material_properties[mask].mean(axis=0)
 
 return self.labels

# Test
materials_data = np.random.randn(50, 5)
clusterer = MaterialsClusterer(n_materials=3)
labels = clusterer.fit(materials_data)

print(f"✓ Materials clustering:")
print(f" Cluster assignments: {labels}")

---