Bayesian Interpretation of Regularization

Throughout our exploration of regularization, we've treated the regularization term as a mathematical constraint—a penalty added to our objective function to prevent overfitting. We've derived closed-form solutions, analyzed shrinkage behavior, and understood the bias-variance tradeoff from an optimization perspective.

But there's a deeper question lurking beneath the surface: Why does adding a penalty term to our loss function work at all? Where do Ridge, Lasso, and Elastic Net penalties come from? Are they arbitrary choices, or is there a principled foundation underlying these techniques?

The answer lies in Bayesian statistics, where regularization emerges naturally from a simple, elegant idea: we have beliefs about our parameters before seeing any data, and we can encode those beliefs mathematically as prior distributions.

Before diving into priors specifically, we need to understand the Bayesian framework for statistical inference. This perspective differs fundamentally from the frequentist approach we've been using.

The Frequentist View:

In frequentist statistics, model parameters $\boldsymbol{\theta}$ are fixed but unknown constants. We estimate them from data, and all probability statements concern the data or estimation procedures, never the parameters themselves. Questions like "What is the probability that $\theta_1 > 0$?" are meaningless—$\theta_1$ is either greater than zero or it isn't.

The Bayesian View:

In Bayesian statistics, we treat parameters $\boldsymbol{\theta}$ as random variables with their own probability distributions. This allows us to:

• Express uncertainty about parameters probabilistically
• Incorporate prior knowledge before seeing data
• Update our beliefs as new data arrives
• Make probabilistic statements about parameters directly

The Core Bayesian Equation:

The entire Bayesian framework rests on Bayes' theorem, applied to parameters and data:

$$p(\boldsymbol{\theta} \mid \mathbf{X}, \mathbf{y}) = \frac{p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\theta}) \cdot p(\boldsymbol{\theta})}{p(\mathbf{y} \mid \mathbf{X})}$$

Let's dissect each component:

The prior distribution $p(\boldsymbol{\theta})$ is arguably the most distinctive element of Bayesian analysis. It encapsulates everything we believe about parameters before observing any data from the current dataset.

What Priors Represent:

Priors can encode various types of information:

1. Domain knowledge: A medical researcher studying drug effects might know that most drugs have small or no effect, encoding this as a prior concentrated near zero.

2. Previous experiments: If prior studies estimated a parameter to be around 0.5 with some uncertainty, this forms a natural prior for new experiments.

3. Physical constraints: If a parameter represents a probability, we know it must be in $[0, 1]$. If it represents a count, it must be non-negative.

4. Regularization preferences: We might believe that simpler models (smaller coefficients) are more likely—this is precisely what regularization priors encode.

Visualizing the Prior's Role:

Imagine estimating the mean height of a new population. Before measuring anyone:

• A vague prior might be $\mathcal{N}(170, 50^2)$—heights around 170cm but with huge uncertainty
• An informative prior might be $\mathcal{N}(175, 10^2)$—based on similar populations studied before
• A skeptical prior might be $\mathcal{N}(170, 5^2)$—strongly believing heights cluster tightly around 170cm

Each prior leads to different posteriors, especially with small samples. As sample size grows, all three converge toward the true population mean—the data eventually overwhelms any reasonable prior.

Prior distributions span a spectrum from completely uninformative to highly specific. Understanding this spectrum is crucial for appreciating how different priors lead to different regularization behaviors.

Flat Priors and Their Limitations:

A flat (or uniform) prior seems appealingly "objective"—we're not biasing the analysis with any assumptions. However, flat priors have serious issues:

1. They're not uninformative: A flat prior on $\theta$ is not a flat prior on $\theta^2$ or $\log(\theta)$. The notion of "no information" depends on parameterization.

2. Improper priors: Flat priors over infinite domains don't integrate to 1. They're "improper" and can sometimes lead to improper posteriors.

3. Poor performance in high dimensions: With many parameters, flat priors become overwhelmingly diffuse, leading to unstable estimates.

4. No regularization effect: Flat priors provide no shrinkage, equivalent to unregularized maximum likelihood—exactly what causes overfitting.

Conjugate Priors:

A conjugate prior is one where the posterior distribution belongs to the same family as the prior. This isn't just mathematically elegant—it enables closed-form solutions.

Common conjugate pairs:

For regression with Gaussian noise and a Gaussian prior on coefficients, the posterior is also Gaussian—this is the foundation of Bayesian linear regression and, as we'll see, Ridge regression.

Our focus is regression, where we estimate coefficients $\boldsymbol{\beta} = (\beta_1, \ldots, \beta_p)^T$. What beliefs might we have about these coefficients before seeing data?

The Case for Shrinkage Priors:

In most regression problems, we have good reasons to believe:

1. Most coefficients are probably small — Not all features matter equally; many have weak or no effect

2. Extremely large coefficients are unlikely — Coefficients exploding to ±100 usually indicates overfitting, not true signal

3. Some coefficients may be exactly zero — In high-dimensional settings, true sparsity is common; many features are irrelevant

These beliefs translate directly into shrinkage priors—distributions that concentrate probability mass near zero while allowing occasional larger values.

Two Fundamental Choices:

The two priors most directly connected to regularization are:

1. Gaussian (Normal) Prior: $$\beta_j \sim \mathcal{N}(0, \tau^2)$$

Places probability symmetrically around zero with decreasing probability for larger values. The density: $$p(\beta_j) = \frac{1}{\sqrt{2\pi\tau^2}} \exp\left(-\frac{\beta_j^2}{2\tau^2}\right)$$

• Smooth, bell-shaped distribution
• Never assigns zero probability to any value
• Leads to Ridge regression (L2 regularization)

2. Laplace (Double Exponential) Prior: $$\beta_j \sim \text{Laplace}(0, b)$$

Also centered at zero but with a sharp peak and heavier tails. The density: $$p(\beta_j) = \frac{1}{2b} \exp\left(-\frac{|\beta_j|}{b}\right)$$

• Sharp peak at zero (more probability mass near zero)
• Heavier tails than Gaussian (allows occasional large values)
• Leads to Lasso regression (L1 regularization)

The Gaussian and Laplace priors, despite both being centered at zero, encode fundamentally different beliefs about coefficient structure. Understanding these differences illuminates why Ridge and Lasso behave so differently.

Geometric Intuition:

To understand the difference geometrically, consider the level sets of each prior (contours of equal probability):

• Gaussian prior: Level sets are circles (in 2D) or hyperspheres. All coefficients shrink proportionally.

• Laplace prior: Level sets are diamonds (in 2D) or cross-polytopes. The corners of the diamond lie on the coordinate axes.

When we combine these priors with the likelihood (which forms elliptical contours for quadratic loss), something remarkable happens:

• The ellipse intersects the circle away from the axes → Ridge estimates are small but non-zero
• The ellipse intersects the diamond at a corner → Lasso estimates are exactly zero for some coefficients

Tail Behavior and Robustness:

The heavier tails of the Laplace prior have an important practical consequence: Laplace priors are more tolerant of occasional large coefficients.

With a Gaussian prior:

• The penalty for a coefficient of size 10 is $10^2 = 100$
• The penalty for two coefficients of size 7.07 is $2 \times 7.07^2 = 100$
• The prior prefers spreading signal across multiple coefficients

With a Laplace prior:

• The penalty for a coefficient of size 10 is $|10| = 10$
• The penalty for two coefficients of size 5 is $2 \times |5| = 10$
• Equal penalty, so large single coefficients are not discouraged

This explains why Ridge tends to distribute effect sizes across correlated predictors, while Lasso often selects one and zeros the others.

We've described Gaussian and Laplace priors and their properties. Now we'll establish the mathematical connection between priors and regularization penalties. This connection is the theoretical foundation for understanding regularization from a Bayesian perspective.

The Key Observation:

The regularization penalty in optimization objectives is the negative log-prior.

For a Gaussian prior $\beta_j \sim \mathcal{N}(0, \tau^2)$: $$\log p(\beta_j) = -\frac{\beta_j^2}{2\tau^2} - \frac{1}{2}\log(2\pi\tau^2)$$

The negative log-prior (ignoring constants) is: $$-\log p(\beta_j) \propto \frac{\beta_j^2}{2\tau^2}$$

Summing over all coefficients: $$-\log p(\boldsymbol{\beta}) \propto \frac{1}{2\tau^2} \sum_{j=1}^p \beta_j^2 = \frac{1}{2\tau^2} |\boldsymbol{\beta}|_2^2$$

This is exactly the L2 penalty with $\lambda = \frac{1}{\tau^2}$!

The Same for Laplace:

For a Laplace prior $\beta_j \sim \text{Laplace}(0, b)$: $$\log p(\beta_j) = -\frac{|\beta_j|}{b} - \log(2b)$$

The negative log-prior is: $$-\log p(\beta_j) \propto \frac{|\beta_j|}{b}$$

Summing over coefficients: $$-\log p(\boldsymbol{\beta}) \propto \frac{1}{b} \sum_{j=1}^p |\beta_j| = \frac{1}{b} |\boldsymbol{\beta}|_1$$

This is exactly the L1 penalty with $\lambda = \frac{1}{b}$!

The Unified View:

We can now see the regularized regression objective in a new light:

$$\text{Loss}(\boldsymbol{\beta}) + \lambda \cdot R(\boldsymbol{\beta}) = -\log p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}) - \log p(\boldsymbol{\beta})$$

$$= -\log\left[p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}) \cdot p(\boldsymbol{\beta})\right]$$

$$= -\log\left[p(\boldsymbol{\beta} \mid \mathbf{X}, \mathbf{y}) \cdot p(\mathbf{y} \mid \mathbf{X})\right]$$

Minimizing the regularized loss is equivalent to maximizing the posterior probability of parameters given data—Maximum A Posteriori (MAP) estimation!

The prior distributions we've discussed have hyperparameters: variance $\tau^2$ for Gaussian, scale $b$ for Laplace. These directly control regularization strength. How should we choose them?

Approaches to Hyperparameter Selection:

Empirical Bayes:

The marginal likelihood approach finds hyperparameters that maximize the probability of the observed data, averaged over all possible parameter values:

$$p(\mathbf{y} \mid \mathbf{X}, \tau^2) = \int p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}) \cdot p(\boldsymbol{\beta} \mid \tau^2) , d\boldsymbol{\beta}$$

For Gaussian priors and Gaussian likelihoods, this integral has a closed form: $$\mathbf{y} \mid \mathbf{X}, \tau^2 \sim \mathcal{N}(\mathbf{0}, \tau^2 \mathbf{X}\mathbf{X}^T + \sigma^2 \mathbf{I})$$

Maximizing this over $\tau^2$ provides a principled, data-driven regularization strength—one that balances model complexity against data fit automatically.

Hierarchical Priors (Full Bayes):

The most thorough approach places priors on hyperparameters:

$$\boldsymbol{\beta} \mid \tau^2 \sim \mathcal{N}(\mathbf{0}, \tau^2 \mathbf{I})$$ $$\tau^2 \sim \text{Inverse-Gamma}(a, b)$$

This fully Bayesian treatment marginalizes over hyperparameter uncertainty, providing more calibrated uncertainty quantification. The cost is increased computational complexity—typically requiring MCMC methods.

So far, we've assumed independent priors across coefficients: $$p(\boldsymbol{\beta}) = \prod_{j=1}^p p(\beta_j)$$

This independence assumption is convenient but not necessary. In many applications, we have prior knowledge about relationships between coefficients.

Correlated Priors:

We can specify a joint prior with non-diagonal covariance: $$\boldsymbol{\beta} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma}_0)$$

where $\boldsymbol{\Sigma}_0$ encodes prior correlations between coefficients.

The Corresponding Regularization:

A Gaussian prior with covariance $\boldsymbol{\Sigma}_0$ yields the penalty: $$R(\boldsymbol{\beta}) = \boldsymbol{\beta}^T \boldsymbol{\Sigma}_0^{-1} \boldsymbol{\beta}$$

This is a generalized Ridge penalty. When $\boldsymbol{\Sigma}_0 = \tau^2 \mathbf{I}$, we recover standard Ridge. But other choices yield different regularization behaviors:

• Smoothing regularization: $\boldsymbol{\Sigma}_0^{-1}$ as a discrete Laplacian encourages smooth coefficient sequences
• Fused regularization: Off-diagonal structure penalizes differences between adjacent coefficients
• Graph regularization: Structure derived from feature similarity graphs

The Bayesian framework makes it clear that choosing a regularization matrix is equivalent to choosing a prior covariance structure.

Understanding priors theoretically is valuable; applying them correctly requires attention to practical details.

Prior Predictive Checks:

A powerful diagnostic is the prior predictive distribution—the distribution of data implied by your prior before seeing actual observations:

1. Sample $\boldsymbol{\beta}$ from the prior $p(\boldsymbol{\beta})$
2. For each sample, generate $\mathbf{y}$ from the likelihood $p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta})$
3. Examine the resulting distribution of $\mathbf{y}$

If your prior implies outcomes like negative heights, probabilities > 1, or incomes of $10 million for entry-level positions, the prior needs adjustment. Good priors should generate plausible (if imprecise) predictions before seeing any data.

We've established the foundational concept of prior distributions and their connection to regularization. This provides the theoretical grounding for the pages that follow.

Key Takeaways:

What's Next:

In the following pages, we'll develop these ideas in depth:

• Page 2: Ridge as Gaussian Prior — The complete mathematical derivation showing Ridge regression as Bayesian inference with Gaussian priors
• Page 3: Lasso as Laplace Prior — Why Laplace priors lead to L1 penalties and sparse solutions
• Page 4: MAP Estimation — The optimization perspective on Bayesian point estimation
• Page 5: Posterior Inference — Moving beyond point estimates to full posterior distributions and uncertainty quantification