Gaussian Processes Introduction

Imagine you're asked to predict the price of a house given its square footage. Traditional approaches would have you choose a specific function—perhaps a line, a polynomial, or a neural network—and then fit parameters to your data. But what if, instead of committing to a single function, you could reason about all possible functions at once?

This is precisely what Gaussian Processes (GPs) offer: a principled probabilistic framework for defining distributions over the space of functions. Rather than learning point estimates of parameters, GPs maintain complete probability distributions that express our uncertainty about which function generated our data.

This shift in perspective—from "find the best function" to "maintain beliefs about all functions"—is subtle but profound. It transforms regression from an optimization problem into an inference problem, bringing the full power of Bayesian reasoning to bear on function estimation.

Before we can appreciate Gaussian Processes, we must understand the paradigm they transcend. In parametric regression, we specify a functional form with a finite number of parameters. Consider the familiar linear model:

$$f(x) = w_0 + w_1 x$$

Here, we've committed to a specific hypothesis class—the set of all linear functions—and our task is to find the optimal parameters $(w_0, w_1)$. This approach has served us well, but it carries fundamental limitations:

The deeper problem:

The parametric approach forces us to compress our beliefs about an infinite-dimensional object (a function) into a finite-dimensional parameter vector. This compression necessarily loses information. Even with Bayesian treatment where we maintain distributions over parameters, we cannot express beliefs about functions outside our chosen hypothesis class.

Consider trying to fit the function $f(x) = \sin(x)$ with a third-degree polynomial. No matter how much data we collect, our model cannot represent the true function—the polynomial hypothesis class simply doesn't contain sinusoids. We've introduced an inductive bias that no data can overcome.

What if we could avoid choosing a functional form entirely?

The conceptual leap that defines Gaussian Processes is elegantly simple: treat the unknown function itself as a random variable. Just as we can define probability distributions over numbers (like the Gaussian distribution over real numbers) or vectors (like the multivariate Gaussian), we can define probability distributions over functions.

But what does this mean, precisely? A function $f: \mathcal{X} \rightarrow \mathbb{R}$ maps inputs to outputs. To treat $f$ as a random variable, we imagine there's a probability distribution $p(f)$ from which functions are "drawn." Some functions have high probability (they're more likely to be the true function); others have low probability.

The key insight: Although functions are infinite-dimensional objects (they assign a value to every point in their domain), we only ever query them at finitely many points. If we care about the function values $f(x_1), f(x_2), \ldots, f(x_n)$ at specific inputs, we need only specify the joint distribution over these finite collections.

Formal definition:

A Gaussian Process is a collection of random variables, any finite number of which have a joint Gaussian distribution. Mathematically, $f \sim \mathcal{GP}(m, k)$ means that for any finite set of inputs ${x_1, x_2, \ldots, x_n}$, the corresponding function values are jointly Gaussian:

$$\begin{pmatrix} f(x_1) \ f(x_2) \ \vdots \ f(x_n) \end{pmatrix} \sim \mathcal{N}\left( \begin{pmatrix} m(x_1) \ m(x_2) \ \vdots \ m(x_n) \end{pmatrix}, \begin{pmatrix} k(x_1, x_1) & k(x_1, x_2) & \cdots & k(x_1, x_n) \ k(x_2, x_1) & k(x_2, x_2) & \cdots & k(x_2, x_n) \ \vdots & \vdots & \ddots & \vdots \ k(x_n, x_1) & k(x_n, x_2) & \cdots & k(x_n, x_n) \end{pmatrix} \right)$$

Here:

• $m(x)$ is the mean function: our prior belief about the expected value of $f$ at any input $x$
• $k(x, x')$ is the covariance function (or kernel): our belief about how function values at different inputs co-vary

The kernel function $k(x, x')$ is the heart of a Gaussian Process. While the mean function captures our prior belief about "where" functions tend to be, the kernel captures our beliefs about the structure and smoothness of functions.

Intuition: The kernel $k(x, x')$ tells us how "similar" the function values at inputs $x$ and $x'$ should be. If $x$ and $x'$ are nearby (however we define "nearby" in input space), and we believe functions tend to be smooth, then $f(x)$ and $f(x')$ should be highly correlated.

Consider the most common kernel—the Squared Exponential (or RBF) kernel:

$$k(x, x') = \sigma^2 \exp\left( -\frac{|x - x'|^2}{2\ell^2} \right)$$

This kernel exhibits key properties:

The kernel encodes our inductive bias:

Unlike parametric models where we choose a functional form (linear, polynomial, etc.), GPs encode assumptions through the kernel. The kernel answers questions like:

• How smooth should the function be?
• Should it have periodic components?
• Are some input dimensions more important than others?
• How does variance change across input space?

This is a fundamentally different way to inject domain knowledge. Rather than saying "the function is a polynomial of degree 3," we say "the function should be smooth, with correlation decaying over a characteristic length scale." The former restricts us to a 4-dimensional family; the latter allows infinite flexibility while still incorporating prior knowledge.

To build intuition about GP priors, let's understand how to draw samples from them. Although a GP defines a distribution over functions (infinite-dimensional objects), we can always evaluate samples at a finite set of points.

The procedure:

1. Choose evaluation points $X = {x_1, x_2, \ldots, x_n}$
2. Compute mean vector $\mathbf{m} = [m(x_1), m(x_2), \ldots, m(x_n)]^T$ (often zero)
3. Compute kernel matrix $K$ where $K_{ij} = k(x_i, x_j)$
4. Sample from multivariate Gaussian $\mathbf{f} \sim \mathcal{N}(\mathbf{m}, K)$

The resulting vector $\mathbf{f}$ gives us function values at the chosen points. By plotting these values and connecting them, we visualize a sample function.

Computational note: Sampling from a multivariate Gaussian requires computing the Cholesky decomposition of $K$: $$\mathbf{f} = \mathbf{m} + L\mathbf{z}$$ where $K = LL^T$ and $\mathbf{z} \sim \mathcal{N}(0, I)$ is a vector of independent standard normals.

Let's formalize the mathematical structure underlying Gaussian Processes. This foundation is essential for understanding posterior inference and making rigorous predictions.

Stochastic Process Definition:

A stochastic process is a collection of random variables indexed by some set. For a GP, this index set is the input domain $\mathcal{X}$ (often $\mathbb{R}^d$). We write ${f(x) : x \in \mathcal{X}}$ to denote this collection.

Gaussian Process Definition (Formal):

A stochastic process ${f(x) : x \in \mathcal{X}}$ is a Gaussian Process if, for every finite collection of indices ${x_1, \ldots, x_n} \subset \mathcal{X}$, the random vector $(f(x_1), \ldots, f(x_n))$ follows a multivariate Gaussian distribution.

This definition is remarkable in its simplicity: we only require that finite marginals be Gaussian. The Kolmogorov extension theorem guarantees that this is sufficient to define a valid probability distribution over functions.

Properties of the Multivariate Gaussian:

To work with GPs, we need facility with multivariate Gaussian distributions. Key properties:

1. Marginalization: If $(\mathbf{f}_1, \mathbf{f}_2)$ is jointly Gaussian, then each marginal is also Gaussian:

$$\begin{pmatrix} \mathbf{f}1 \ \mathbf{f}2 \end{pmatrix} \sim \mathcal{N}\left( \begin{pmatrix} \boldsymbol{\mu}1 \ \boldsymbol{\mu}2 \end{pmatrix}, \begin{pmatrix} \Sigma{11} & \Sigma{12} \ \Sigma{21} & \Sigma{22} \end{pmatrix} \right)$$

implies

$$\mathbf{f}_1 \sim \mathcal{N}(\boldsymbol{\mu}1, \Sigma{11})$$

2. Conditioning: The conditional distribution of $\mathbf{f}_1$ given $\mathbf{f}_2$ is also Gaussian:

$$\mathbf{f}1 | \mathbf{f}2 \sim \mathcal{N}(\boldsymbol{\mu}{1|2}, \Sigma{1|2})$$

where:

• $\boldsymbol{\mu}{1|2} = \boldsymbol{\mu}1 + \Sigma{12}\Sigma{22}^{-1}(\mathbf{f}_2 - \boldsymbol{\mu}_2)$
• $\Sigma_{1|2} = \Sigma_{11} - \Sigma_{12}\Sigma_{22}^{-1}\Sigma_{21}$

These formulas are the engine of GP inference. When we observe data, we condition the GP on those observations to obtain a posterior distribution over functions.

A natural question arises: how can we work with infinite-dimensional objects (functions) using finite-dimensional tools (linear algebra)? The answer lies in the consistency of GP marginals.

The consistency principle:

Suppose we have observed function values at points ${x_1, \ldots, x_n}$. If we later query the function at a new point $x_{n+1}$, the GP gives us a joint distribution over ${f(x_1), \ldots, f(x_{n+1})}$. The crucial property is that the marginal distribution over the original $n$ points is exactly what the GP specified before—adding new query points doesn't change predictions for old points.

This consistency means we can:

• Define distributions at arbitrary resolution
• Add or remove evaluation points without contradictions
• Make predictions at any new location by computing the appropriate conditional

The function space perspective:

While we implement GPs using finite-dimensional linear algebra, it helps to visualize the underlying infinite-dimensional structure. Consider the space of all continuous functions $C(\mathcal{X})$ equipped with a suitable topology. A GP defines a probability measure on this space such that:

• Smooth functions (as encoded by the kernel) have high probability
• Functions that violate our prior assumptions have low probability
• The measure is consistent across all finite-dimensional projections

Given that we want to define distributions over functions, why specifically Gaussian? Several compelling reasons make the Gaussian choice natural and practical:

1. Analytical Tractability:

Gaussian distributions are the only distributions that are closed under both marginalization and conditioning. This means:

• Every marginal of a Gaussian is Gaussian
• Every conditional of a Gaussian is Gaussian
• Linear combinations of Gaussians are Gaussian

For GPs, this closedness is essential. We need to marginalize (ignore some function values) and condition (observe some function values) constantly. Any other distribution family would require approximations.

2. Maximum Entropy:

For a fixed mean and covariance, the Gaussian distribution has maximum entropy. This means it's the "least informative" distribution consistent with our specified first and second moments. If we only want to encode beliefs about mean and covariance structure, Gaussian is the principled choice—we're not injecting additional assumptions.

3. Central Limit Theorem:

Under mild conditions, sums of many independent random variables converge to Gaussians. Since many real-world quantities arise as aggregates of many small effects, Gaussian models are often reasonable approximations to reality.

4. Computational Efficiency:

Working with Gaussians involves linear algebra operations: matrix multiplication, inversion, and Cholesky decomposition. These operations are:

• Well-understood mathematically
• Numerically stable with appropriate techniques
• Efficiently implemented in every scientific computing library
• Easily parallelized on modern hardware

Contrast this with distributions requiring numerical integration or sampling—Gaussians admit closed-form solutions.

5. Complete Specification:

A Gaussian distribution is fully characterized by its first two moments—mean and covariance. Higher moments are determined by these. This means specifying a GP requires only:

• A mean function $m(x)$
• A kernel function $k(x, x')$

We don't need to specify third moments (skewness), fourth moments (kurtosis), or any higher-order structure. The kernel alone captures everything about the distribution's shape.

To develop intuition about GP priors, let's visualize the distribution over functions as confidence bands, not just individual samples.

At each input $x$, the GP prior gives us a marginal distribution for $f(x)$. With zero mean and kernel $k$:

$$f(x) \sim \mathcal{N}(0, k(x, x)) = \mathcal{N}(0, \sigma^2)$$

This marginal tells us: before seeing any data, each function value is distributed around zero with variance $\sigma^2$. But this marginal alone doesn't capture the correlation structure—for that, we need to examine how function values at different inputs co-vary.

Confidence bands:

For a GP prior $f \sim \mathcal{GP}(0, k)$, we can plot:

• The mean function: $m(x) = 0$ (a flat line at zero)
• The pointwise uncertainty: $\pm 2\sqrt{k(x,x)} = \pm 2\sigma$

These bands represent where 95% of function values fall at each point, assuming uncorrelated marginals. The bands don't capture correlation—they just show marginal uncertainty.

We've explored the fundamental concept underlying Gaussian Processes: treating functions as random variables drawn from probability distributions. This perspective shift enables principled uncertainty quantification and flexible nonparametric modeling.

What's Next:

With the foundation of GP priors established, we're ready to tackle the next crucial component: prior specification. We'll examine how to choose mean functions and kernels that encode appropriate domain knowledge, exploring the rich space of kernel designs and their implications for modeling.