Gaussian Process

A Gaussian process (GP) is a probability distribution over functions, defined by the property that the function's values at any finite collection of input points are jointly normally distributed; it is specified entirely by a mean function and a covariance function, called the kernel, that says how strongly the function values at two inputs are expected to move together. Used as a prior over unknown functions, it yields a flexible, nonparametric Bayesian regression and classification method that returns, for every query point, not just a prediction but a predictive distribution with an honest uncertainty. The standard reference is Carl Edward Rasmussen and Christopher Williams's book "Gaussian Processes for Machine Learning" (MIT Press, 2006). The same mathematics was developed in geostatistics, where it is called kriging, after Danie Krige and Georges Matheron's formalization.

The kernel carries all the modeling assumptions: it encodes smoothness, scale, periodicity and trend, and choosing or combining kernels is the main act of design in a GP. The squared-exponential (RBF) kernel gives infinitely smooth functions with a length-scale controlling how quickly correlation decays; the Matérn family gives rougher, more realistic sample paths and is often preferred for physical data; periodic kernels capture seasonality; linear kernels recover Bayesian linear regression. Sums and products of kernels combine these effects, as in the long-term trend plus seasonal plus noise models popularized in the book. Kernel hyperparameters (length-scale, signal variance, noise variance) are usually fitted by maximizing the marginal likelihood, which balances data fit against model complexity automatically and is closely related to the machinery of maximum-likelihood-estimation applied after integrating out the function itself.

Prediction is exact and closed-form because conditioning a joint Gaussian on some of its components gives another Gaussian, which makes GPs one of the few models where Bayesian inference needs no approximation for regression. The posterior mean is a kernel-weighted combination of the training outputs, the posterior covariance shrinks near observed points and reverts to the prior far from them, and this automatic widening of error bars away from the data is the feature that makes GPs valuable when uncertainty matters. The price is computational: exact inference needs a matrix inversion or Cholesky factorization of an n × n kernel matrix, costing O(n³) time and O(n²) memory, which limits exact GPs to thousands or tens of thousands of points. Sparse approximations with inducing points (Quiñonero-Candela and Rasmussen, 2005; Titsias, 2009), structured kernels and GPU-based libraries such as GPyTorch and GPflow, plus scikit-learn's implementation, push the limit much higher. For classification the likelihood is not Gaussian, so the posterior must be approximated with Laplace or variational methods.

Bayesian optimization is the most visible modern use of GPs: when each evaluation is expensive (training a neural network, running a wet-lab experiment, tuning a chip design), a GP surrogate guides which point to try next. The surrogate's mean and uncertainty are combined by an acquisition function such as expected improvement, used in Jones, Schonlau and Welch's 1998 "Efficient Global Optimization" method, or the upper confidence bound analyzed by Srinivas and colleagues in 2010 as a bandit problem. Snoek, Larochelle and Adams's 2012 paper "Practical Bayesian Optimization of Machine Learning Algorithms" brought the approach into hyperparameter tuning, and tools such as Spearmint, BoTorch and Ax followed. GPs also connect to deep learning: Radford Neal showed in 1996 that a one-hidden-layer network with infinitely many units and random weights converges to a GP, and later work extended this to deep networks and to the neural tangent kernel picture.

The standard pitfalls are treating the kernel as a detail, overtrusting the uncertainty, and ignoring scale. A GP is only as good as its kernel: a stationary smooth kernel cannot represent a sudden change, and extrapolation simply returns to the prior mean with wide error bars, which is honest but rarely a useful forecast. Predictive uncertainty is conditional on the kernel and on the hyperparameters being right, and fitting hyperparameters by marginal likelihood can overfit with little data or leave multiple local optima. In Bayesian optimization, performance is commonly reported to degrade beyond a few tens of dimensions unless the problem has low effective dimensionality, a failure of distance-based kernels in high-dimensional space. Compared with a neural network, a GP is data-efficient but not scalable; compared with the weight-space Bayesian treatment of a parametric model, it places the prior on the function directly, which is easier to reason about than a prior on millions of weights.

Read a Gaussian process as a prior over entire functions that learns by conditioning on a handful of points, with the kernel stating what kind of function is plausible, the predictive variance stating how far to trust it, and the cubic cost reminding that this elegance belongs to small, expensive-data problems.