Mixture Models and Clustering for Data Analysis
# Mixture Models and Clustering for Data Analysis
## Introduction & Motivation
Mixture models probabilistically cluster data, assigning samples to latent components. Essential for materials characterization, process identification, customer segmentation, and unsupervised discovery in scientific and engineering applications.
Motivation: Use mixture models for principled clustering and component discovery.
Applications: Clustering, density estimation, component identification, anomaly detection.
---
## Core Concepts & Theory
### Mixture Components
Latent data clusters.
### Soft Assignment
Probabilistic membership.
### EM Algorithm
Parameter estimation.
### Model Selection
Choosing component count.
---
## Mathematical Formulation
Gaussian Mixture:
$$p(x) = \sum_{k=1}^K \pi_k \mathcal{N}(x|\mu_k, \Sigma_k)$$
Posterior Responsibility:
$$\gamma(z_k|x) = \frac{\pi_k \mathcal{N}(x|\mu_k, \Sigma_k)}{\sum_j \pi_j \mathcal{N}(x|\mu_j, \Sigma_j)}$$
Log-Likelihood:
$$\ln p(X| heta) = \sum_n \ln \left\{ \sum_k \pi_k \mathcal{N}(x_n|\mu_k, \Sigma_k)
ight\}$$
---
## Advanced Theory & Extensions
### Mixture of Experts
Gating networks for components.
### Hierarchical Clustering
Tree-structured mixtures.
### Variational Bayes
Approximate posteriors.
---
## Computational Considerations
E-Step: O(N·K·D).
M-Step: O(N·K·D²).
Convergence: O(I·N·K·D²) for I iterations.
---
## Practical Implementation Strategies
### Initialization
K-means or random starts.
### Convergence Criteria
Log-likelihood threshold.
### Regularization
Preventing singular covariances.
---
## Benchmark Datasets & Evaluation
Iris Dataset: Multivariate clustering.
MNIST Handwritten Digits: Image clustering.
Materials Data: Property clustering.
---
## Key Challenges & Limitations
### Model Selection
Determining K.
### Local Optima
EM convergence issues.
### Degeneracy
Singular covariances.
---
## Hyperparameter Tuning
Components: 2-10.
Covariance type: Spherical, diagonal, full.
Regularization: 1e-6 to 1e-3.
---
## Real-World Applications & Case Studies
Materials: Phase identification.
Sensors: Fault mode clustering.
Chemistry: Reaction outcome classification.
---
## Integration with Other Methods
Mixture models + Bayesian inference; + clustering; + density estimation.
---
## Summary & Key Takeaways
Mixture models enable probabilistic clustering.
Principles:
1. Model: Define mixture components.
2. Assignment: Compute posteriors.
3. Training: EM optimization.
4. Selection: Choose component count.
5. Application: Solve clustering tasks.
---
## Appendix: Practical Labs
### Lab 1: Gaussian Mixture Model
import numpy as np
class GaussianMixture:
def __init__(self, n_components=3):
self.n_components = n_components
self.means = None
self.covariances = None
self.weights = None
def initialize(self, X):
"""Initialize parameters"""
n_samples, n_features = X.shape
# Random initialization
indices = np.random.choice(n_samples, self.n_components, replace=False)
self.means = X[indices].copy()
self.covariances = [np.eye(n_features) for _ in range(self.n_components)]
self.weights = np.ones(self.n_components) / self.n_components
def e_step(self, X):
"""Compute responsibilities"""
n_samples = len(X)
responsibilities = np.zeros((n_samples, self.n_components))
for k in range(self.n_components):
diff = X - self.means[k]
cov = self.covariances[k] + np.eye(X.shape[1]) * 1e-6
det = np.linalg.det(cov)
inv = np.linalg.inv(cov)
mahal = np.sum(diff @ inv * diff, axis=1)
responsibilities[:, k] = self.weights[k] * np.exp(-0.5 * mahal) / np.sqrt(det)
responsibilities /= (responsibilities.sum(axis=1, keepdims=True) + 1e-10)
return responsibilities
def m_step(self, X, responsibilities):
"""Update parameters"""
n_samples = len(X)
for k in range(self.n_components):
Nk = responsibilities[:, k].sum()
self.weights[k] = Nk / n_samples
self.means[k] = (X * responsibilities[:, k, None]).sum(axis=0) / (Nk + 1e-10)
diff = X - self.means[k]
self.covariances[k] = (diff.T * responsibilities[:, k]) @ diff / (Nk + 1e-10)
# Test
gmm = GaussianMixture(n_components=3)
X = np.vstack([np.random.randn(30, 2), np.random.randn(30, 2) + 3])
gmm.initialize(X)
resp = gmm.e_step(X)
gmm.m_step(X, resp)
print(f"✓ Gaussian mixture model:")
print(f" Weights: {gmm.weights}")### Lab 2: EM Algorithm
import numpy as np
def em_clustering(X, n_components=3, n_iter=100):
"""EM algorithm"""
n_samples, n_features = X.shape
# Initialize
means = X[np.random.choice(n_samples, n_components, replace=False)]
weights = np.ones(n_components) / n_components
covariances = [np.eye(n_features) for _ in range(n_components)]
for iteration in range(n_iter):
# E-step
responsibilities = np.zeros((n_samples, n_components))
for k in range(n_components):
diff = X - means[k]
cov = covariances[k] + np.eye(n_features) * 1e-6
inv_cov = np.linalg.inv(cov)
det_cov = np.linalg.det(cov)
mahal = np.sum(diff @ inv_cov * diff, axis=1)
responsibilities[:, k] = weights[k] * np.exp(-0.5 * mahal) / np.sqrt(det_cov + 1e-10)
responsibilities /= responsibilities.sum(axis=1, keepdims=True) + 1e-10
# M-step
for k in range(n_components):
Nk = responsibilities[:, k].sum()
weights[k] = Nk / n_samples
means[k] = (X * responsibilities[:, k, None]).sum(axis=0) / (Nk + 1e-10)
return means, weights, responsibilities
X = np.random.randn(100, 2)
means, weights, resp = em_clustering(X, n_components=2, n_iter=50)
print(f"✓ EM clustering convergence achieved")### Lab 3: Model Selection
import numpy as np
def bic_criterion(X, n_components, ll):
"""Bayesian Information Criterion"""
n_samples, n_features = X.shape
n_params = n_components * (2 * n_features + 1) - 1
bic = -2 * ll + n_params * np.log(n_samples)
return bic
def aic_criterion(X, n_components, ll):
"""Akaike Information Criterion"""
n_features = X.shape[1]
n_params = n_components * (2 * n_features + 1) - 1
aic = -2 * ll + 2 * n_params
return aic
print(f"✓ Model selection criteria configured")### Lab 4: Clustering Application
import numpy as np
class MaterialsClusterer:
def __init__(self, n_materials=3):
self.n_components = n_materials
self.means = None
self.labels = None
def fit(self, material_properties):
"""Cluster materials"""
n_samples, n_features = material_properties.shape
# Initialize means
self.means = material_properties[np.random.choice(n_samples, self.n_components, replace=False)]
# Simple k-means-like clustering
for _ in range(10):
distances = np.zeros((n_samples, self.n_components))
for k in range(self.n_components):
distances[:, k] = np.linalg.norm(material_properties - self.means[k], axis=1)
self.labels = np.argmin(distances, axis=1)
# Update means
for k in range(self.n_components):
mask = self.labels == k
if mask.sum() > 0:
self.means[k] = material_properties[mask].mean(axis=0)
return self.labels
# Test
materials_data = np.random.randn(50, 5)
clusterer = MaterialsClusterer(n_materials=3)
labels = clusterer.fit(materials_data)
print(f"✓ Materials clustering:")
print(f" Cluster assignments: {labels}")---