The Bayesian Framework

Imagine you're a medical researcher investigating a new diagnostic test for a rare disease. Before collecting any data, you already possess valuable knowledge: the disease affects approximately 1 in 10,000 people in the general population. This knowledge—prior to observing any test results—fundamentally shapes how you should interpret any evidence you subsequently gather.

This is the essence of prior beliefs in Bayesian inference: the formal mathematical encoding of what we know (or believe) about a phenomenon before observing new data. Unlike frequentist statistics, which treats parameters as fixed but unknown constants, Bayesian inference treats parameters as random variables with probability distributions that capture our uncertainty about their true values.

Before diving into mathematical formalism, we must understand why priors exist and what they represent. The concept of prior beliefs emerges from a fundamental philosophical position about the nature of probability itself.

Two Interpretations of Probability:

In the frequentist interpretation, probability represents the long-run frequency of events in repeated experiments. Under this view, parameters are fixed constants—there's nothing probabilistic about them. The only randomness comes from sampling variability.

In the Bayesian (subjectivist) interpretation, probability represents a degree of belief about uncertain propositions. Under this view, probability quantifies our state of knowledge, and parameters are random variables whose distributions encode our uncertainty.

This distinction is profound: Bayesians can assign probabilities to any uncertain proposition, including:

• "The coin's bias is between 0.4 and 0.6"
• "The treatment effect is positive"
• "Model A is more appropriate than Model B"

These statements are meaningless in frequentist probability because they concern fixed quantities, not random events. But in Bayesian probability, they represent coherent expressions of belief.

Historical Context:

The Reverend Thomas Bayes (1701–1761) first formulated the theorem bearing his name in a posthumously published essay. However, the modern conception of subjective probability was developed extensively by Bruno de Finetti, Leonard Savage, and others in the 20th century.

De Finetti's famous theorem demonstrates that exchangeability (the assumption that the order of observations doesn't matter) implies the existence of a prior distribution over parameters. This provides a rigorous foundation for Bayesian inference that doesn't require accepting subjective probability as a philosophical primitive—it emerges naturally from more basic modeling assumptions.

The Coherence Argument:

Another powerful justification for Bayesian probability comes from Dutch book arguments. If your beliefs don't satisfy the axioms of probability (including Bayes' theorem for updating), then there exist sequences of bets that guarantee you lose money regardless of outcomes. Only probability-coherent beliefs avoid such exploitability.

This means Bayesian reasoning isn't merely one approach among many—it's the unique coherent approach to reasoning under uncertainty.

Having established the philosophical foundations, let's formalize prior beliefs mathematically. A prior distribution is a probability distribution over the parameter space that encodes our beliefs before observing data.

Notation and Basic Setup:

Let θ (theta) denote the unknown parameter(s) of our model. The prior distribution is denoted:

$$p(\theta)$$

This is a probability density function (for continuous parameters) or probability mass function (for discrete parameters) that satisfies the standard requirements:

• Non-negativity: $p(\theta) \geq 0$ for all $\theta$
• Normalization: $\int p(\theta) d\theta = 1$ (or $\sum_{\theta} p(\theta) = 1$ for discrete)

The choice of $p(\theta)$ reflects our prior knowledge. More probability mass in a region means we believe values in that region are more plausible a priori.

The Parameter Space:

The form of the prior depends critically on the parameter space—the set of all possible values the parameter can take:

The prior must have support matching the parameter space. Using a Normal prior for a probability parameter would be invalid because Normal distributions place mass on negative values and values greater than 1.

Multivariate Priors:

When models have multiple parameters θ = (θ₁, θ₂, ..., θₖ), we specify a joint prior distribution:

$$p(\theta) = p(\theta_1, \theta_2, ..., \theta_k)$$

A common simplification is to assume prior independence:

$$p(\theta) = \prod_{i=1}^{k} p(\theta_i)$$

This factorization makes prior specification tractable but may not always be appropriate. Sometimes parameters have known relationships that should be encoded in the prior structure.

For example, in a hierarchical model where individual effects are drawn from a population distribution:

• Individual effects: $\theta_i \sim \mathcal{N}(\mu, \sigma^2)$
• Population mean: $\mu \sim \mathcal{N}(0, \tau^2)$
• Population variance: $\sigma^2 \sim \text{InverseGamma}(\alpha, \beta)$

Here, the θᵢ are conditionally independent given μ and σ², but marginally they are correlated through their shared hyperparameters.

Priors can be classified along several dimensions. The most important classification concerns their informativeness—how much constraint they place on parameter values. Understanding this taxonomy is essential for appropriate prior selection.

Let's examine each category in depth.

Non-informative Priors:

The quest for 'objective' priors that represent complete ignorance has a long history. The most naive approach is the flat (uniform) prior:

$$p(\theta) \propto 1$$

This seems intuitive—all values are equally likely. However, flat priors have serious problems:

1. Non-invariance under reparameterization: If $p(\theta) \propto 1$, then for $\phi = g(\theta)$, the prior on $\phi$ is NOT flat: $p(\phi) \propto |d\theta/d\phi|$

2. Impropriety: On unbounded spaces, flat priors have infinite integral—they're not proper probability distributions

3. Information paradox: A flat prior on [0, 1] for probability θ is actually informative—it implies specific beliefs about log-odds or other transformations

Jeffreys Prior:

Harold Jeffreys proposed a principled solution: the prior should be proportional to the square root of the Fisher Information:

$$p(\theta) \propto \sqrt{I(\theta)}$$

where $I(\theta) = -E\left[\frac{\partial^2}{\partial\theta^2}\log p(x|\theta)\right]$ is the Fisher Information.

Jeffreys priors are transformation-invariant: the same prior is obtained regardless of how the parameter is expressed. For a Bernoulli likelihood, Jeffreys prior is Beta(1/2, 1/2)—not uniform!

Weakly Informative Priors:

Modern Bayesian practice, particularly as advocated by Andrew Gelman and collaborators, emphasizes weakly informative priors as a practical default. These priors:

• Rule out clearly impossible or implausible values
• Have minimal impact when data is abundant
• Provide regularization when data is sparse
• Are interpretable and defensible

For example, when modeling a regression coefficient β representing the effect of a standardized predictor on a standardized outcome:

$$\beta \sim \mathcal{N}(0, 1)$$

asserts that effects larger than ±2 standard deviations are unlikely (95% prior probability between -2 and 2). This is weak enough to let data dominate with reasonable sample sizes, but provides regularization against overfitting.

Informative Priors from Domain Knowledge:

When genuine prior knowledge exists, it should be used. Examples include:

• Physics: Physical constants are known precisely from prior experiments
• Medicine: Drug effect sizes from related compounds or populations
• Economics: Historical data on inflation, interest rates, etc.
• Meta-analysis: Summary of previous studies provides a natural informative prior

Encoding this knowledge as an informative prior is a strength of Bayesian inference, not a weakness. It formalizes the scientific process of building on prior work.

Certain distribution families appear repeatedly in Bayesian modeling due to their mathematical convenience and flexibility. Mastering these families is essential for effective prior specification.

The Beta Distribution in Detail:

The Beta distribution is perhaps the most important prior family for beginners. It's the natural prior for parameters representing probabilities or proportions.

$$p(\theta | \alpha, \beta) = \frac{\Gamma(\alpha + \beta)}{\Gamma(\alpha)\Gamma(\beta)} \theta^{\alpha-1}(1-\theta)^{\beta-1}$$

Interpreting α and β:

• Think of α-1 as "prior successes" and β-1 as "prior failures" (pseudocounts)
• Mean: E[θ] = α/(α+β)
• Mode (for α,β > 1): (α-1)/(α+β-2)
• Concentration: Larger α+β means more concentrated (less uncertain)

Special cases:

• Beta(1, 1) = Uniform(0, 1): Complete ignorance
• Beta(0.5, 0.5): Jeffreys prior
• Beta(2, 2): Weakly informative, favors middle values
• Beta(10, 10): Informative, strongly favors values near 0.5
• Beta(1, 10): Favors small values
• Beta(10, 1): Favors large values

The Normal Distribution for Location Parameters:

For unbounded real-valued parameters, the Normal (Gaussian) distribution is a natural choice:

$$p(\theta | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(\theta - \mu)^2}{2\sigma^2}\right)$$

The Normal distribution has maximum entropy among all distributions with a given mean and variance, making it a principled default when only these moments are known.

Guidelines for Normal priors:

• Center μ at your best guess for the parameter
• Set σ large enough to cover plausible values
• Remember: 95% of mass falls within μ ± 1.96σ
• For standardized regression coefficients, σ = 1 or σ = 2.5 are common defaults

Half-Cauchy for Scale Parameters:

For parameters that must be positive and represent scales or standard deviations, the half-Cauchy distribution is increasingly recommended:

$$p(\sigma | s) = \frac{2}{\pi s \left(1 + (\sigma/s)^2\right)}, \quad \sigma > 0$$

The half-Cauchy has heavier tails than Gamma or Inverse-Gamma alternatives, providing robustness when the true scale is unexpectedly large. This is particularly valuable in hierarchical models where shrinkage might otherwise be excessive.

Selecting appropriate priors is both an art and a science. Here we present a systematic framework for making principled prior choices.

Encoding Domain Knowledge:

Suppose you're modeling the probability θ that a patient responds to a new cancer treatment. From the literature, similar drugs in this class have response rates between 15% and 35%, with a best estimate around 25%.

Step 1: What prior encodes this?

We want a Beta(α, β) with:

• Mean ≈ 0.25: So α/(α+β) ≈ 0.25, meaning α ≈ β/3
• 95% of mass between 0.15 and 0.35

Using numerical methods or trial-and-error, Beta(10, 30) satisfies these constraints:

• Mean = 10/40 = 0.25 ✓
• 2.5th percentile ≈ 0.14, 97.5th percentile ≈ 0.38 ≈ Close enough

Alternatively, Beta(5, 15) gives similar mean but wider spread—appropriate if literature is less certain.

An improper prior is a function that does not integrate to a finite value—it's not a valid probability distribution. The most common example is the flat prior on an unbounded space:

$$p(\theta) \propto 1, \quad \theta \in (-\infty, \infty)$$

Since $\int_{-\infty}^{\infty} 1 , d\theta = \infty$, this is not a proper probability distribution.

Why use improper priors?

Despite their theoretical issues, improper priors are sometimes used because:

1. They can yield proper posteriors when the likelihood is informative enough
2. They represent a form of 'letting the data speak'
3. They can simplify computation in certain cases

The Danger:

Improper priors can lead to improper posteriors—posteriors that also don't integrate to finite values. This makes inference impossible: you can't compute means, credible intervals, or any meaningful summaries.

Example of disaster:

Consider a hierarchical model:

• Observations: $y_i \sim \mathcal{N}(\mu, \sigma^2)$ for $i = 1, ..., n$
• Prior on mean: $p(\mu) \propto 1$ (improper flat prior)
• Prior on variance: $p(\sigma^2) \propto 1/\sigma^2$ (improper Jeffreys-style prior)

If n = 1 (single observation), the posterior on σ² is improper—there's no finite integral. The model is unidentified and inference is meaningless.

Safe practices:

1. Always verify the posterior is proper before using improper priors
2. Use proper weakly informative priors as a safer default
3. If using improper priors, document the theoretical justification
4. With MCMC, improper posteriors may not be obvious—chains may appear to converge but give misleading results

A critical aspect of responsible Bayesian analysis is understanding how much conclusions depend on prior assumptions. Sensitivity analysis systematically varies the prior and observes how the posterior changes.

When to worry:

• With abundant data, posteriors should be robust to reasonable prior variations—the likelihood dominates
• With limited data, posteriors may be sensitive to prior choice—this indicates our conclusions are prior-driven
• Extreme sensitivity suggests either very weak data or substantively different priors encoding genuinely different assumptions

Formal sensitivity analysis:

1. Specify a class of priors: Define a range of 'reasonable' priors that all could be justified
2. Compute posterior under each: Fit the model with each prior
3. Report range of conclusions: If all reasonable priors yield similar conclusions, results are robust
4. Identify influential priors: Understand which aspects of the prior (location? spread? tail behavior?) drive sensitivity

Interpreting Sensitivity:

In the example above with only 10 observations:

• Posterior means range from approximately 0.25 to 0.38 depending on prior
• This spread of 0.13 indicates moderate sensitivity
• With more data (e.g., 30 successes in 100 trials), posteriors would converge regardless of prior

The Asymptotics of Prior Influence:

As sample size n → ∞, the posterior converges to the true parameter value regardless of the prior (assuming the model is well-specified and the prior places non-zero mass on the true value). This is called posterior consistency.

The rate of convergence is typically $O(1/\sqrt{n})$, meaning the prior's influence shrinks proportionally. After 100 observations, the prior has roughly 1/10th the influence it had after 1 observation.

This provides theoretical comfort: with enough data, prior misspecification is automatically corrected. But in finite samples, prior choice matters.

Prior beliefs form the foundation of Bayesian inference—they encode what we know before observing data. Let's consolidate the key concepts:

What's Next:

With prior beliefs established, we now turn to the likelihood function—the mathematical bridge connecting our model's parameters to observed data. The likelihood quantifies how probable our observations are under different parameter values, and its combination with the prior yields the posterior distribution.