OneNoughtOne
0/328
Loading content...
Read aloud is unavailable in this browser.
Bayesian inference provides a mathematically principled framework for updating beliefs in light of evidence. At its heart lies Bayes' theorem, the elegant formulation that transforms prior beliefs into posterior conclusions:
$$P(\theta | D) = \frac{P(D | \theta) \cdot P(\theta)}{P(D)}$$
where $\theta$ represents our model parameters, $D$ is the observed data, $P(\theta)$ encodes our prior beliefs, $P(D|\theta)$ is the likelihood of data given parameters, and $P(\theta|D)$ is the posterior distribution we seek.
The fundamental challenge of Bayesian inference is not conceptual—it's computational. The denominator $P(D) = \int P(D|\theta) P(\theta) d\theta$ requires integrating over the entire parameter space. For most realistic models, this integral is analytically intractable. We cannot write down a closed-form expression for the posterior.
This computational barrier stood as a major obstacle to practical Bayesian methods for centuries. The breakthrough came through a beautiful mathematical property: conjugacy. Conjugate priors transform the intractable into the elegant, allowing exact posterior computation for an important class of models.
By the end of this page, you will understand the formal definition of conjugate priors, the mathematical conditions that create conjugacy, why this property matters for practical inference, and the taxonomy of conjugate families used throughout Bayesian machine learning. You'll develop the intuition to recognize when conjugacy applies and when approximation methods become necessary.
The concept of conjugacy formalizes a specific algebraic relationship between prior and likelihood distributions that guarantees computational tractability.
Definition (Conjugate Prior): A family of prior distributions $\mathcal{F}$ is said to be conjugate to a likelihood function $P(D|\theta)$ if, whenever the prior $P(\theta)$ belongs to $\mathcal{F}$, the posterior distribution $P(\theta|D)$ also belongs to $\mathcal{F}$.
More precisely, if $P(\theta | \eta) \in \mathcal{F}$ where $\eta$ represents the hyperparameters of the prior, then there exists a deterministic function $\phi$ such that:
$$P(\theta | D) = P(\theta | \eta') \in \mathcal{F}, \quad \text{where } \eta' = \phi(\eta, D)$$
The posterior is simply another member of the same family, differing only in its hyperparameters. The function $\phi$ specifies exactly how the data $D$ updates the prior hyperparameters $\eta$ to posterior hyperparameters $\eta'$.
Conjugacy is fundamentally a closure property. Just as rational numbers are closed under addition (adding two rationals gives a rational), conjugate prior families are closed under Bayesian updating. The posterior remains within the family, no matter how much data we observe.
Why this definition matters computationally:
Without conjugacy, computing the posterior requires:
With conjugacy, all three challenges vanish:
This transformation—from integral computation to parameter updates—is what makes conjugacy so powerful.
| Aspect | Without Conjugacy | With Conjugacy |
|---|---|---|
| Posterior form | Unknown functional form | Same family as prior |
| Normalization | Intractable integral | Analytic (known from family) |
| Computation | MCMC, variational, etc. | Closed-form update equations |
| Representation | Samples or approximation | Finite hyperparameters |
| Sequential updates | Recompute from scratch | Update hyperparameters incrementally |
The existence of conjugate priors is not accidental—it emerges from the mathematical structure of the exponential family of distributions. Understanding this connection reveals when conjugacy exists and what form conjugate priors take.
Definition (Exponential Family): A probability distribution belongs to the exponential family if its density can be written as:
$$P(x | \theta) = h(x) \cdot \exp\left( \eta(\theta)^T T(x) - A(\theta) \right)$$
where:
The sufficient statistic $T(x)$ is central to conjugacy. It compresses all data into a fixed-dimensional summary that loses no information about $\theta$. For $n$ observations, we don't need $n$ values—we need only the aggregate sufficient statistics. This compression is what enables tractable Bayesian updating.
Theorem (Conjugate Prior Existence): For any likelihood from the exponential family, a conjugate prior exists and has the form:
$$P(\theta | \tau, u) \propto \exp\left( \eta(\theta)^T \tau - u \cdot A(\theta) \right)$$
where:
The posterior, after observing data $x_1, \ldots, x_n$, has hyperparameters:
$$\tau' = \tau + \sum_{i=1}^n T(x_i), \quad u' = u + n$$
This is remarkably elegant: the prior hyperparameters encode pseudo-observations, and updating simply involves adding the sufficient statistics of real observations.
Mathematical Derivation:
Let's derive why the posterior remains in the conjugate family. Starting from Bayes' theorem:
$$P(\theta | x_{1:n}) \propto P(x_{1:n} | \theta) \cdot P(\theta)$$
For i.i.d. data from an exponential family:
$$P(x_{1:n} | \theta) = \prod_{i=1}^n h(x_i) \cdot \exp\left( \eta(\theta)^T \sum_{i=1}^n T(x_i) - n \cdot A(\theta) \right)$$
Multiplying by the conjugate prior:
$$P(\theta | x_{1:n}) \propto \exp\left( \eta(\theta)^T \left[ \tau + \sum_{i=1}^n T(x_i) \right] - ( u + n) \cdot A(\theta) \right)$$
This has exactly the form of the conjugate prior with updated hyperparameters $\tau' = \tau + \sum_{i=1}^n T(x_i)$ and $ u' = u + n$.
The proof is complete: the posterior is in the same family as the prior. ∎
One of the most powerful conceptual tools for working with conjugate priors is the pseudo-observation interpretation. This framework provides intuition for choosing hyperparameters and understanding their effect on inference.
The Core Insight: Hyperparameters in a conjugate prior encode imaginary prior observations. The prior behaves as if we had already seen some data before the actual experiment.
Consider a simple example. In the Beta-Binomial model:
The hyperparameters $\alpha$ and $\beta$ behave as if:
After observing real data ($k$ successes, $n - k$ failures), we simply add them to our pseudo-observations.
Think of the prior pseudo-count $ u$ as the 'weight' of the prior. A prior with $ u = 2$ (based on 2 pseudo-observations) will be easily overwhelmed by data. A prior with $ u = 1000$ (based on 1000 pseudo-observations) requires substantial evidence to shift significantly. This provides intuitive control over prior-data balance.
| Model | Prior Hyperparameters | Pseudo-Observation Interpretation |
|---|---|---|
| Beta-Binomial | $\alpha, \beta$ | $\alpha - 1$ successes, $\beta - 1$ failures |
| Dirichlet-Multinomial | $\alpha_1, \ldots, \alpha_K$ | $\alpha_k - 1$ observations in category $k$ |
| Gamma-Poisson | $\alpha, \beta$ | $\alpha$ total events in time $\beta$ |
| Normal-Normal (known variance) | $\mu_0, \sigma_0^2$ | Sample mean $\mu_0$ from $n_0 = \sigma^2 / \sigma_0^2$ observations |
| Normal-Inverse-Gamma | $\mu_0, \kappa, \alpha, \beta$ | Sample mean $\mu_0$ from $\kappa$ obs; $2\alpha$ degrees of freedom for variance |
Practical Guidelines for Setting Hyperparameters:
The pseudo-observation interpretation directly guides hyperparameter selection:
1. Weak (Vague) Priors: Set the pseudo-count to be small relative to expected data size. For Beta-Binomial:
These priors allow data to dominate almost immediately.
2. Informative Priors: Translate domain knowledge into equivalent observations. If you believe the success rate is around 80%:
3. Skeptical Priors: When testing interventions, encode skepticism that they work:
With small datasets, the prior substantially affects conclusions. Always perform sensitivity analysis: how do conclusions change with different reasonable priors? If conclusions are sensitive, report this uncertainty honestly. Conjugate priors make such sensitivity analysis computationally trivial—just change hyperparameters and recompute.
One of the most practically valuable properties of conjugate priors is their natural support for sequential (online) learning. When data arrives incrementally, conjugate priors allow efficient updates without storing or reprocessing historical data.
The Sequential Property: For conjugate priors, the posterior after observing data $D_1$ followed by data $D_2$ is identical to the posterior after observing all data simultaneously:
$$P(\theta | D_1, D_2) = P(\theta | D_1 \cup D_2)$$
Moreover, we can compute this sequentially:
$$\eta_2 = \phi(\eta_1, D_2) = \phi(\phi(\eta_0, D_1), D_2)$$
Each update requires only the current hyperparameters and new data—not the full history.
Real-World Applications of Sequential Updating:
1. Online A/B Testing: As users interact with variants A and B, we can continuously update Beta posteriors:
No need to wait for a fixed sample size—the Bayesian posterior is valid at every step.
2. Streaming Data Analysis: For IoT sensors, log streams, or real-time feeds:
3. Memory-Constrained Systems: Edge devices, embedded systems, or applications with privacy constraints:
4. Hierarchical Models: In hierarchical Bayesian models:
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081
import numpy as np class BetaBinomialTracker: """ Online Bayesian tracker for success probability. Uses conjugate Beta-Binomial model for O(1) updates. """ def __init__(self, alpha_prior: float = 1.0, beta_prior: float = 1.0): """ Initialize with prior hyperparameters. Args: alpha_prior: Prior pseudo-successes (default: uniform prior) beta_prior: Prior pseudo-failures (default: uniform prior) """ self.alpha = alpha_prior self.beta = beta_prior self.n_observations = 0 def update(self, successes: int, trials: int = 1) -> None: """ Update posterior with new observations. Computational complexity: O(1) regardless of history size. Memory complexity: O(1) - only hyperparameters stored. """ self.alpha += successes self.beta += trials - successes self.n_observations += trials def posterior_mean(self) -> float: """Expected value of success probability under posterior.""" return self.alpha / (self.alpha + self.beta) def posterior_mode(self) -> float: """Mode of posterior (MAP estimate).""" if self.alpha > 1 and self.beta > 1: return (self.alpha - 1) / (self.alpha + self.beta - 2) return self.posterior_mean() # Fallback if mode undefined def credible_interval(self, confidence: float = 0.95) -> tuple: """ Equal-tailed credible interval for success probability. """ from scipy import stats tail = (1 - confidence) / 2 lower = stats.beta.ppf(tail, self.alpha, self.beta) upper = stats.beta.ppf(1 - tail, self.alpha, self.beta) return (lower, upper) def prob_greater_than(self, threshold: float) -> float: """P(θ > threshold | data), useful for decision making.""" from scipy import stats return 1 - stats.beta.cdf(threshold, self.alpha, self.beta) # Example: Online A/B testingtracker_A = BetaBinomialTracker(alpha_prior=1, beta_prior=1)tracker_B = BetaBinomialTracker(alpha_prior=1, beta_prior=1) # Simulate streaming dataobservations = [ ('A', 1, 1), # User 1: variant A, converted ('B', 0, 1), # User 2: variant B, did not convert ('A', 1, 1), # User 3: variant A, converted ('B', 1, 1), # User 4: variant B, converted # ... continues with streaming data] for variant, success, trials in observations: if variant == 'A': tracker_A.update(success, trials) else: tracker_B.update(success, trials) # At any point, we can compute current inference print(f"After {tracker_A.n_observations + tracker_B.n_observations} obs:") print(f" P(A better) = {compute_prob_a_beats_b(tracker_A, tracker_B):.3f}") print(f" A: {tracker_A.posterior_mean():.3f} {tracker_A.credible_interval()}") print(f" B: {tracker_B.posterior_mean():.3f} {tracker_B.credible_interval()}")The landscape of conjugate priors is rich and varied, with each family suited to different data types and inference tasks. Understanding this taxonomy equips you to select appropriate models across diverse applications.
Organizing Principle: Conjugate families can be organized by the type of data they model and the parameters being estimated. The following comprehensive table serves as a reference for practical Bayesian modeling.
| Likelihood | Conjugate Prior | Posterior | Application Domain |
|---|---|---|---|
| Bernoulli($p$) | Beta($\alpha, \beta$) | Beta($\alpha + k, \beta + n - k$) | Binary outcomes, click rates, conversion |
| Binomial($n, p$) | Beta($\alpha, \beta$) | Beta($\alpha + k, \beta + n - k$) | Success counts, A/B testing |
| Multinomial($p_1, \ldots, p_K$) | Dirichlet($\alpha_1, \ldots, \alpha_K$) | Dirichlet($\alpha_k + n_k$) | Category distributions, topic models |
| Poisson($\lambda$) | Gamma($\alpha, \beta$) | Gamma($\alpha + \sum x_i, \beta + n$) | Count data, event rates, queuing |
| Exponential($\lambda$) | Gamma($\alpha, \beta$) | Gamma($\alpha + n, \beta + \sum x_i$) | Waiting times, survival analysis |
| Normal($\mu$, known $\sigma^2$) | Normal($\mu_0, \sigma_0^2$) | Normal($\mu_n, \sigma_n^2$) | Continuous measurements |
| Normal(known $\mu$, $\sigma^2$) | Inverse-Gamma($\alpha, \beta$) | Inverse-Gamma updated | Variance estimation |
| Normal($\mu, \sigma^2$) both unknown | Normal-Inverse-Gamma | Normal-Inverse-Gamma | Full location-scale inference |
| Multivariate Normal($\mu$, known $\Sigma$) | Multivariate Normal | Multivariate Normal | Vector observations |
| Multivariate Normal(known $\mu$, $\Sigma$) | Inverse-Wishart | Inverse-Wishart | Covariance matrix estimation |
| Multivariate Normal($\mu, \Sigma$) | Normal-Inverse-Wishart | Normal-Inverse-Wishart | Full multivariate inference |
For exponential family likelihoods, the conjugate prior described by the exponential family theory (Section 2) is called the natural conjugate prior. While there may be other priors with conjugate-like properties, the natural conjugate is uniquely determined by the likelihood's sufficient statistics and provides the most elegant computational framework.
Hierarchical Extensions:
The conjugate families above extend naturally to hierarchical models:
1. Beta-Binomial Hierarchical Model:
2. Dirichlet-Multinomial Hierarchical Model:
3. Normal-Inverse-Gamma Hierarchical Model:
These hierarchical extensions preserve much of the computational convenience of conjugacy while modeling realistic multi-level structure.
While conjugate priors provide elegant solutions, they have fundamental limitations that practitioners must understand. Recognizing when conjugacy fails is as important as knowing when it applies.
Fundamental Limitation: Conjugacy is likelihood-specific. A prior family is conjugate to a specific likelihood. Change the likelihood, and conjugacy may break. This creates a tension between model flexibility and computational convenience.
Don't let computational convenience drive model choice. If a conjugate prior poorly represents your actual beliefs, using it introduces bias that no amount of data will correct. Better to use approximate inference with a realistic prior than exact inference with a misleading one.
Alternatives When Conjugacy Fails:
When conjugate priors are unsuitable, modern Bayesian computation offers several alternatives:
1. Markov Chain Monte Carlo (MCMC):
2. Variational Inference:
3. Laplace Approximation:
4. Expectation Propagation:
5. Semi-Conjugate Analysis:
| Method | Accuracy | Speed | Best For |
|---|---|---|---|
| MCMC | Asymptotically exact | Slow (hours to days) | Research, final analysis, complex models |
| Variational | Approximate (mean-field bias) | Fast (minutes to hours) | Large datasets, deep models, exploration |
| Laplace | Gaussian approximation | Very fast (seconds) | Large n, quick estimates |
| EP | Good for skewed posteriors | Moderate | Classification, sparse models |
Conjugate priors have a rich history intertwined with the development of Bayesian statistics itself. Understanding this context illuminates why certain conventions exist and how the field has evolved.
Historical Development:
1763 - Thomas Bayes: Published (posthumously) the first conjugate analysis—using a uniform (Beta(1,1)) prior for a binomial proportion. Bayes recognized that the posterior had the same functional form as the prior.
1812 - Pierre-Simon Laplace: Systematized Bayesian methods and extensively used Beta-Binomial conjugacy. Laplace's "rule of succession" ($\frac{k+1}{n+2}$ for estimating a probability) derives from a uniform Beta prior.
1930s-1950s - Harold Jeffreys: Developed non-informative prior theory, identifying the Jeffreys prior for various models. For many exponential family models, Jeffreys priors are proper conjugate priors with specific hyperparameters.
1960s-1970s - Formal Conjugacy Theory: Raiffa and Schlaifer (1961) provided the first systematic treatment of conjugate priors. Diaconis and Ylvisaker (1979) proved that only exponential family likelihoods admit finite-dimensional conjugate priors.
1980s-Present - Computational Revolution: MCMC methods (Gibbs sampling, Metropolis-Hastings) reduced reliance on conjugacy. Yet conjugate priors remain valuable for fast approximate inference, initialization, and interpretability.
Despite advances in approximate inference, conjugate priors remain central to modern machine learning. Latent Dirichlet Allocation uses Dirichlet-Multinomial conjugacy. Bayesian neural network priors often use Gaussian conjugacy for computational efficiency. Thompson sampling for bandits exploits Beta-Binomial conjugacy for exploration. Understanding conjugacy is not just historical—it's practically essential.
Philosophical Perspectives:
Conjugate priors occupy an interesting philosophical position in Bayesian statistics:
The Subjectivist View: Priors should represent genuine beliefs. If your prior belief happens to be in a conjugate family, wonderful—computation is easy. If not, forcing conjugacy introduces unjustified assumptions. "Let the prior speak for itself."
The Objective Bayesian View: Seek "reference" or "non-informative" priors that minimize the prior's influence on inference. Many non-informative priors (Jeffreys, reference priors) happen to be conjugate, providing both objectivity and tractability.
The Pragmatic View: Conjugate priors are convenient approximations. For large datasets, the prior's influence diminishes, so the tradeoff of slight prior misspecification for computational tractability is worthwhile. "All priors are wrong; some are useful."
The Modern Synthesis: Use conjugate priors when they're appropriate, approximate methods when they're not. Computational considerations are valid but shouldn't dominate scientific validity. Sensitivity analysis reveals when the choice matters.
We have explored the mathematical foundations and practical implications of conjugate priors—a cornerstone of tractable Bayesian inference. Let us consolidate the key insights:
What's Next:
Having established the formal foundation, the following pages will explore specific conjugate families in depth:
Each family reveals its own insights and applications, building toward fluent Bayesian practice.
You now understand the formal definition of conjugate priors, their mathematical foundation in exponential family theory, the pseudo-observation interpretation of hyperparameters, and the broader context in which conjugacy operates. Next, we'll dive deep into the Beta-Binomial conjugacy—the most widely used conjugate family in practice.