expectation-maximization and variants

# Expectation-Maximization and Variants

## Introduction & Motivation

EM algorithm maximizes likelihood for models with latent variables. Fundamental for clustering, density estimation, and latent factor discovery in engineering and scientific applications with incomplete data.

Motivation: Learn with unobserved variables via EM.

Applications: Clustering, parameter estimation, latent factor analysis, incomplete data modeling.

---

## Core Concepts & Theory

### E-Step

Posterior estimation of latent variables.

### M-Step

Maximum likelihood parameter update.

### Convergence

Monotonic likelihood increase.

### Missing Data

Handling incomplete observations.

---

## Mathematical Formulation

E-Step:
$$Q( heta| heta^{(t)}) = \mathbb{E}_{z|x, heta^{(t)}}[\ln p(x,z| heta)]$$

M-Step:
$$ heta^{(t+1)} = \arg\max_ heta Q( heta| heta^{(t)})$$

Likelihood Lower Bound:
$$\ln p(x| heta) \geq Q( heta| heta^{(t)}) - H(q)$$

---

## Advanced Theory & Extensions

### Variational EM

Approximate posterior.

### Stochastic EM

Scalable online updates.

### Generalized EM

Partial M-step.

---

## Computational Considerations

E-Step: O(N·K).

M-Step: O(N·K·D).

Convergence: O(I·N·K·D) for I iterations.

---

## Practical Implementation Strategies

### Initialization

Smart starting parameters.

### Convergence Criteria

Log-likelihood threshold.

### Acceleration

Conjugate gradient in M-step.

---

## Benchmark Datasets & Evaluation

Iris Dataset: Multi-component clustering.

MNIST: Digit recognition.

Synthetic Data: Known ground truth.

---

## Key Challenges & Limitations

### Local Optima

Multiple solutions.

### Initialization Sensitivity

Starting point matters.

### Computational Cost

Iterative refinement.

---

## Hyperparameter Tuning

Components: 2-10.

Convergence tolerance: 1e-6 to 1e-3.

Max iterations: 100-1000.

---

## Real-World Applications & Case Studies

Sensor Data: Latent mode identification.

Medical Records: Missing data imputation.

Customer Segmentation: Behavioral clustering.

---

## Integration with Other Methods

EM + mixture models; + latent variable models; + Bayesian inference.

---

## Summary & Key Takeaways

EM enables learning with latent variables.

Principles:
1. E-Step: Posterior inference.
2. M-Step: Parameter update.
3. Iteration: Convergence.
4. Missing: Handle incompleteness.
5. Applications: Latent discovery.

---

## Appendix: Practical Labs

### Lab 1: EM for Gaussian Mixture

import numpy as np

def em_iteration(X, means, covariances, weights):
 """One EM iteration"""
 N, D = X.shape
 K = len(means)
 
 # E-step
 responsibilities = np.zeros((N, K))
 for k in range(K):
 diff = X - means[k]
 cov_inv = np.linalg.inv(covariances[k] + np.eye(D) * 1e-6)
 det_cov = np.linalg.det(covariances[k] + np.eye(D) * 1e-6)
 
 mahal = np.sum(diff @ cov_inv * diff, axis=1)
 responsibilities[:, k] = weights[k] * np.exp(-0.5 * mahal) / np.sqrt(det_cov)
 
 responsibilities /= (responsibilities.sum(axis=1, keepdims=True) + 1e-10)
 
 # M-step
 Nk = responsibilities.sum(axis=0)
 weights = Nk / N
 
 for k in range(K):
 means[k] = (X * responsibilities[:, k:k+1]).sum(axis=0) / (Nk[k] + 1e-10)
 
 return means, covariances, weights, responsibilities

X = np.random.randn(100, 2)
means = X[np.random.choice(len(X), 3)]
covariances = [np.eye(2) for _ in range(3)]
weights = np.ones(3) / 3

for _ in range(10):
 means, covariances, weights, resp = em_iteration(X, means, covariances, weights)

print(f"✓ EM converged: weights {weights}")

### Lab 2: Missing Data Imputation

import numpy as np

def impute_with_em(X_incomplete, max_iter=50):
 """Impute missing values via EM"""
 X = X_incomplete.copy()
 
 # Initial imputation
 for j in range(X.shape[1]):
 mask = np.isnan(X[:, j])
 if mask.sum() > 0:
 X[mask, j] = np.nanmean(X[:, j])
 
 # EM iterations
 for iteration in range(max_iter):
 # E-step: estimate missing
 means = np.nanmean(X, axis=0)
 
 for j in range(X.shape[1]):
 mask = np.isnan(X_incomplete[:, j])
 if mask.sum() > 0:
 X[mask, j] = means[j]
 
 return X

X_full = np.random.randn(50, 3)
X_incomplete = X_full.copy()
X_incomplete[np.random.rand(50, 3) > 0.8] = np.nan

X_imputed = impute_with_em(X_incomplete)
print(f"✓ Missing data imputed: shape {X_imputed.shape}")

### Lab 3: Latent Factor Model

import numpy as np

class LatentFactorEM:
 def __init__(self, n_factors=3):
 self.n_factors = n_factors
 self.loadings = None
 self.factors = None
 
 def fit(self, X, n_iter=10):
 """Fit latent factor model"""
 N, D = X.shape
 
 # Initialize
 self.factors = np.random.randn(N, self.n_factors)
 self.loadings = np.random.randn(D, self.n_factors)
 noise_var = 1.0
 
 for iteration in range(n_iter):
 # E-step: update factors
 self.factors = X @ self.loadings / (np.linalg.norm(self.loadings, axis=0)**2 + noise_var)
 
 # M-step: update loadings
 self.loadings = X.T @ self.factors / (N + 1e-8)
 
 return self

model = LatentFactorEM(n_factors=3)
X = np.random.randn(50, 10)
model.fit(X, n_iter=10)

print(f"✓ Latent factors learned: shape {model.factors.shape}")

### Lab 4: Model Selection

import numpy as np

def compute_likelihood(X, means, covariances, weights):
 """Compute log-likelihood"""
 ll = 0
 for k in range(len(means)):
 diff = X - means[k]
 cov = covariances[k] + np.eye(X.shape[1]) * 1e-6
 cov_inv = np.linalg.inv(cov)
 det_cov = np.linalg.det(cov)
 
 mahal = np.sum(diff @ cov_inv * diff, axis=1)
 ll += weights[k] * np.exp(-0.5 * mahal) / np.sqrt(det_cov)
 
 return np.mean(np.log(ll + 1e-10))

def model_selection_bic(X, K_values):
 """Select K via BIC"""
 bics = []
 
 for K in K_values:
 means = X[np.random.choice(len(X), K)]
 covariances = [np.eye(X.shape[1]) for _ in range(K)]
 weights = np.ones(K) / K
 
 ll = compute_likelihood(X, means, covariances, weights)
 n_params = K * (X.shape[1] + 1)
 bic = -2 * ll + n_params * np.log(len(X))
 bics.append(bic)
 
 return K_values[np.argmin(bics)]

X = np.random.randn(100, 5)
best_K = model_selection_bic(X, [2, 3, 4, 5])
print(f"✓ Best K: {best_K}")

---