Gaussian Mixture Model

A Gaussian mixture model (GMM) is a probabilistic model that describes data as drawn from a weighted blend of several normal-distribution components, each with its own mean and covariance, where which component produced a given point is hidden. The density is a weighted sum: the mixing weights are non-negative and add to one, and each component contributes a bell-shaped (or, in several dimensions, ellipsoidal) bump. By stacking enough such bumps a GMM can approximate almost any smooth density, so it serves at once as a density estimator, a soft-clustering method and a building block inside larger systems. Karl Pearson fitted a mixture of two normal components to measurements of Naples crab populations in 1894, one of the earliest recorded uses of a mixture model.

The central idea is that a point does not belong to one cluster but has a probability of belonging to each, called its responsibility, which is the posterior probability of the component given the point. Given component k with weight π_k, mean μ_k and covariance Σ_k, the responsibility of k for point x is π_k × N(x | μ_k, Σ_k), divided by the sum of the same quantity over all components, a direct application of bayes-theorem. Points near a boundary between two components get fractional memberships, which is the key difference from k-means, whose hard assignments and implicitly equal spherical clusters are the limiting case of a GMM with identical, shrinking, spherical covariances. A GMM can therefore describe elongated, tilted, differently sized clusters, and it reports how unsure it is about borderline points.

Fitting is done almost always by expectation-maximization, which alternates computing responsibilities and re-estimating each component's parameters as responsibility-weighted averages. The updates are closed-form: a new weight is the average responsibility, a new mean is the responsibility-weighted mean, and a new covariance is the responsibility-weighted scatter matrix. The alternative is direct gradient-based maximum-likelihood-estimation, or a fully Bayesian fit in which weights get a Dirichlet prior and the number of components can be inferred (Dirichlet-process mixtures and variational Bayesian GMMs, offered by scikit-learn, do so). Practical libraries expose a covariance structure choice, typically full, tied, diagonal or spherical, to trade flexibility against parameter count: a full covariance in d dimensions has about d² / 2 parameters per component, which becomes unaffordable for high-dimensional data.

GMMs have a long record in applied systems. In speech technology, GMMs over acoustic features formed the emission model of GMM-HMM speech recognizers for decades before deep neural networks replaced them, and Douglas Reynolds and colleagues' GMM-UBM approach (2000) made GMMs the standard for speaker verification. In computer vision, Stauffer and Grimson's 1999 adaptive background model used a mixture of Gaussians per pixel to separate moving objects from a changing scene. They are common in anomaly detection (low density under the fitted mixture is flagged as unusual), in initializing and modeling clusters in genomics and astronomy, in representing the output distribution of mixture density networks, and in the prior models of some generative systems. The number of components is usually selected with the Bayesian information criterion (Schwarz, 1978), whose penalty on the parameter count discourages unnecessary bumps.

The common pitfalls are over-interpreting components as real groups, singular solutions and label switching. A component is a modeling device, not a discovered species: a skewed or heavy-tailed population will often be fit by several Gaussians that merely approximate its shape, so "three components" does not mean "three kinds of thing." If a component's mean collapses onto one point its covariance shrinks to zero and the likelihood tends to infinity, so libraries add a small regularization constant to the diagonal. The likelihood has many local maxima, so multiple restarts or k-means initialization are standard. In Bayesian fits the component labels are interchangeable, so the posterior is multimodal merely through relabeling, and naive averaging across samples misleads. Finally, a GMM assumes the clusters are roughly ellipsoidal; for crescent-shaped or ring-shaped data, density-based or spectral methods fit better.

Read a Gaussian mixture model as a committee of simple bell curves, each claiming part of the data with a stated probability, a flexible and honest way to describe lumpy data as long as its components are treated as approximating tools and not as discoveries.