Gaussian Processes Kernel Methods Non-Parametric Regression
# Gaussian Processes: Kernel Methods & Non-parametric Regression
## Introduction & Motivation
Gaussian Processes: non-parametric Bayesian model. Prior over functions; updated by observations. Kernel defines function similarity. Applications: regression, uncertainty quantification, Bayesian optimization.
Motivation: Flexible function modeling with uncertainty.
Applications: Regression, optimization, exploration.
---
## Core Concepts & Theory
### Gaussian Distribution
Multivariate normal over function values.
### Kernel Function
Defines covariance structure; similarity.
### Posterior Conditioning
Update prior given observations.
---
## Mathematical Formulation
GP definition:
$$f(x) \sim \mathcal{GP}(m(x), k(x, x'))$$
Posterior distribution:
$$p(f_* | x_*, X, y) = \mathcal{N}(\mu_*, \Sigma_*)$$
Predictive mean:
$$\mu_* = K_*^T (K + \sigma_n^2 I)^{-1} y$$
Predictive variance:
$$\Sigma_* = k(x_*, x_*) - K_*^T (K + \sigma_n^2 I)^{-1} K_*$$
---
## Advanced Theory & Extensions
### Sparse GPs
Inducing points; scalability.
### Multi-task GP
Shared kernel structure.
### Deep Kernel Learning
Neural network kernel.
---
## Computational Considerations
Inference: O(N³) matrix inversion.
Prediction: O(N²) per test point.
Sparse GP: O(M²N) with M inducing points.
---
## Practical Implementation Strategies
### Kernel Selection
RBF, Matern, periodic; domain-specific.
### Hyperparameter Learning
Marginal likelihood maximization.
### Numerical Stability
Cholesky decomposition; jitter.
---
## Benchmark Datasets & Evaluation
1D Synthetic: Easy visualization.
UCI Regression: Standard benchmarks.
Bayesian Optimization: Tuning benchmark.
---
## Key Challenges & Limitations
### Computational Cost
O(N³) prohibits large datasets.
### Kernel Choice
Sensitive to kernel selection.
### Extrapolation
Uncertainty grows far from data.
---
## Hyperparameter Tuning
Kernel lengthscale: Data-dependent; affects smoothness.
Noise variance: Measurement uncertainty.
Signal variance: Function amplitude.
---
## Real-World Applications & Case Studies
Bayesian Optimization: Hyperparameter tuning.
Uncertainty Quantification: Confidence intervals.
Sequential Design: Active sampling.
---
## Integration with Other Methods
GP + Neural Network → hybrid model.
GP + Ensemble → improved predictions.
---
## Summary & Key Takeaways
Gaussian Processes via kernel methods enable non-parametric Bayesian regression with principled uncertainty quantification.
Principles:
1. Kernel: function similarity.
2. Prior: GP distribution.
3. Posterior: conditioning on data.
4. Inference: covariance inversion.
5. Scalability: sparse approximations.
---
---
## Appendix: Practical Labs
### Lab 1: Kernel Functions
import numpy as np
def rbf_kernel(X1, X2, lengthscale=1.0):
"""RBF (Squared Exponential) kernel"""
# Euclidean distances
distances_sq = np.sum((X1[:, None, :] - X2[None, :, :]) ** 2, axis=2)
# RBF kernel
K = np.exp(-distances_sq / (2 * lengthscale ** 2))
return K
# Test
np.random.seed(42)
X1 = np.random.randn(10, 3)
X2 = np.random.randn(8, 3)
K = rbf_kernel(X1, X2)
assert K.shape == (10, 8), "Kernel shape"
assert np.all(K >= 0) and np.all(K <= 1), "Kernel in [0,1]"
print("✓ RBF kernel working")
if __name__ == "__main__":
print("Lab 1: RBFKernel - PASSED")### Lab 2: GP Posterior
import numpy as np
def gp_posterior(X_train, y_train, X_test, kernel_fn, sigma_n=0.01):
"""Compute GP posterior distribution"""
# Training kernel
K = kernel_fn(X_train, X_train)
K_noisy = K + sigma_n ** 2 * np.eye(len(X_train))
# Test kernels
K_star = kernel_fn(X_test, X_train)
K_star_star = kernel_fn(X_test, X_test)
# Solve linear system
L = np.linalg.cholesky(K_noisy)
alpha = np.linalg.solve(L.T, np.linalg.solve(L, y_train))
# Posterior mean
mu = K_star @ alpha
# Posterior variance
v = np.linalg.solve(L, K_star.T)
sigma = K_star_star - (v.T @ v)
return mu, sigma
# Test
np.random.seed(42)
X_train = np.random.randn(10, 2)
y_train = np.random.randn(10)
X_test = np.random.randn(5, 2)
def rbf_kernel(X1, X2, lengthscale=1.0):
distances_sq = np.sum((X1[:, None, :] - X2[None, :, :]) ** 2, axis=2)
return np.exp(-distances_sq / (2 * lengthscale ** 2))
mu, sigma = gp_posterior(X_train, y_train, X_test, rbf_kernel)
assert mu.shape == (5,), "Mean shape"
assert sigma.shape == (5, 5), "Covariance shape"
print("✓ GP posterior working")
if __name__ == "__main__":
print("Lab 2: GPPosterior - PASSED")### Lab 3: Marginal Likelihood
import numpy as np
def gp_marginal_likelihood(X, y, kernel_fn, sigma_n=0.01):
"""Compute log marginal likelihood"""
# Covariance matrix
K = kernel_fn(X, X)
K_noisy = K + sigma_n ** 2 * np.eye(len(X))
# Cholesky decomposition
L = np.linalg.cholesky(K_noisy)
# Log determinant
log_det = 2 * np.sum(np.log(np.diag(L)))
# Quadratic term
alpha = np.linalg.solve(L.T, np.linalg.solve(L, y))
quad_term = y @ alpha
# Marginal likelihood
log_likelihood = -0.5 * (quad_term + log_det + len(y) * np.log(2 * np.pi))
return log_likelihood
# Test
np.random.seed(42)
X = np.random.randn(10, 2)
y = np.random.randn(10)
def rbf_kernel(X1, X2, lengthscale=1.0):
distances_sq = np.sum((X1[:, None, :] - X2[None, :, :]) ** 2, axis=2)
return np.exp(-distances_sq / (2 * lengthscale ** 2))
ll = gp_marginal_likelihood(X, y, rbf_kernel)
assert np.isfinite(ll), "Log-likelihood finite"
print("✓ Marginal likelihood working")
if __name__ == "__main__":
print("Lab 3: MarginalLikelihood - PASSED")### Lab 4: Prediction Uncertainty
import numpy as np
def gp_predict_interval(X_train, y_train, X_test, kernel_fn, sigma_n=0.01, confidence=0.95):
"""Compute prediction intervals"""
# GP posterior
K = kernel_fn(X_train, X_train)
K_noisy = K + sigma_n ** 2 * np.eye(len(X_train))
K_star = kernel_fn(X_test, X_train)
K_star_star = kernel_fn(X_test, X_test)
L = np.linalg.cholesky(K_noisy)
alpha = np.linalg.solve(L.T, np.linalg.solve(L, y_train))
mu = K_star @ alpha
v = np.linalg.solve(L, K_star.T)
var = np.diag(K_star_star) - np.sum(v ** 2, axis=0)
std = np.sqrt(np.maximum(var, 0))
# Confidence interval (assuming Gaussian)
z_score = 1.96 if confidence == 0.95 else 1.0
lower = mu - z_score * std
upper = mu + z_score * std
return mu, lower, upper
# Test
np.random.seed(42)
X_train = np.random.randn(10, 2)
y_train = np.random.randn(10)
X_test = np.random.randn(5, 2)
def rbf_kernel(X1, X2, lengthscale=1.0):
distances_sq = np.sum((X1[:, None, :] - X2[None, :, :]) ** 2, axis=2)
return np.exp(-distances_sq / (2 * lengthscale ** 2))
mu, lower, upper = gp_predict_interval(X_train, y_train, X_test, rbf_kernel)
assert mu.shape == (5,), "Mean shape"
assert lower.shape == (5,), "Lower bound shape"
assert np.all(lower <= mu) and np.all(mu <= upper), "Valid intervals"
print("✓ Prediction intervals working")
if __name__ == "__main__":
print("Lab 4: PredictionIntervals - PASSED")