Gaussian Naive Bayes

In our exploration of Naive Bayes, we have encountered powerful techniques for discrete features: Multinomial Naive Bayes excels at text classification by modeling word frequencies, while Bernoulli Naive Bayes handles binary feature vectors elegantly. But what happens when our features are continuous—real-valued measurements like height, temperature, stock prices, or pixel intensities?

Consider a medical diagnosis system that must classify patients based on blood pressure (120.5 mmHg), body temperature (37.2°C), and cholesterol level (195 mg/dL). These are not word counts or binary flags—they are measurements from a continuous spectrum. How do we compute $P(\text{blood_pressure} = 120.5 | \text{disease})$? The probability of observing any exact real number is technically zero under continuous distributions.

This fundamental challenge motivates Gaussian Naive Bayes, where we model each continuous feature as following a Gaussian (normal) distribution within each class. This elegant assumption transforms an impossible probability computation into a tractable density estimation problem.

Let us first understand precisely why the techniques we developed for Multinomial and Bernoulli Naive Bayes cannot be directly applied to continuous features.

Discrete Feature Modeling: A Review

For discrete features (e.g., word counts in text), we estimate class-conditional probabilities by counting:

$$P(x_j = v | y = k) = \frac{\text{count of samples in class } k \text{ where feature } j = v}{\text{total samples in class } k}$$

This works because:

1. Features take values from a finite set ${v_1, v_2, \ldots, v_m}$
2. Multiple samples can have the exact same value
3. Probabilities are additive: $\sum_{v} P(x_j = v | y = k) = 1$

The Continuous Feature Problem

For continuous features, these assumptions break down:

Problem 1: Infinite possible values

A feature like height can take any value in a continuous range (e.g., 150.00000... to 200.00000... cm). There are uncountably infinite possible values, not a finite set.

Problem 2: Zero probability of exact values

In a continuous distribution, the probability of observing any exact value is zero: $$P(X = 175.3247281...) = 0$$

This is because probability mass must be distributed over infinitely many possible values.

Problem 3: Counting doesn't work

If we try to estimate $P(\text{height} = 175.0 | \text{male})$ by counting, we might find that out of 1000 male samples, exactly zero have height precisely equal to 175.0 (maybe 174.98 or 175.02, but not exactly 175.0). Our probability estimate would be zero, which is both mathematically correct and practically useless.

The mathematical tool for handling continuous random variables is the probability density function (PDF), which fundamentally differs from the probability mass functions (PMFs) used for discrete variables.

From Mass to Density

Probability Mass Function (PMF) — for discrete variables:

• $p(x)$ gives the exact probability that $X = x$
• $\sum_x p(x) = 1$
• Individual values have non-zero probability

Probability Density Function (PDF) — for continuous variables:

• $f(x)$ gives the density at point $x$, not probability
• $\int_{-\infty}^{\infty} f(x) dx = 1$
• Individual points have zero probability; only intervals have non-zero probability

$$P(a \leq X \leq b) = \int_a^b f(x) dx$$

Interpreting Density

The density $f(x)$ indicates the relative likelihood of observing values near $x$. If $f(x_1) = 2 \cdot f(x_2)$, then observing a value near $x_1$ is twice as likely as observing a value near $x_2$.

Crucial insight: While $P(X = x) = 0$ for any specific $x$, the density $f(x)$ can be large or small, telling us where values are more or less likely to concentrate.

$$f(x) = \lim_{\epsilon \to 0} \frac{P(x - \epsilon < X < x + \epsilon)}{2\epsilon}$$

The density is the limit of probability per unit length as the interval shrinks to zero.

The elegant insight that enables Gaussian Naive Bayes is that we can substitute probability densities for probability mass in Bayes' theorem, and the classification still works correctly.

Standard Bayes' Theorem (Discrete)

For discrete features: $$P(y = k | \mathbf{x}) = \frac{P(\mathbf{x} | y = k) P(y = k)}{P(\mathbf{x})}$$

We classify by finding: $$\hat{y} = \arg\max_k P(y = k | \mathbf{x}) = \arg\max_k P(\mathbf{x} | y = k) P(y = k)$$

Modified Bayes' Theorem (Continuous)

For continuous features, we replace probabilities with probability densities: $$f(y = k | \mathbf{x}) \propto f(\mathbf{x} | y = k) P(y = k)$$

Where:

• $f(\mathbf{x} | y = k)$ is the class-conditional density of feature vector $\mathbf{x}$ given class $k$
• $P(y = k)$ is the prior probability of class $k$ (still a probability, not density)

We classify by finding: $$\hat{y} = \arg\max_k f(\mathbf{x} | y = k) P(y = k)$$

Why This Works

The key insight is that we are comparing ratios of likelihoods across classes. Since we're comparing: $$\frac{f(\mathbf{x} | y = 1) P(y = 1)}{f(\mathbf{x} | y = 0) P(y = 0)}$$

Any constants that don't depend on class $k$ cancel out. The densities serve as relative likelihoods—telling us which class makes the observed data more likely.

Mathematical Justification

The formal justification involves considering infinitesimally small regions around the observed point. For a small region $\mathcal{B}$ around $\mathbf{x}$:

$$P(y = k | \mathbf{x} \in \mathcal{B}) = \frac{P(\mathbf{x} \in \mathcal{B} | y = k) P(y = k)}{P(\mathbf{x} \in \mathcal{B})}$$

As $\mathcal{B}$ shrinks to a point: $$P(\mathbf{x} \in \mathcal{B} | y = k) \approx f(\mathbf{x} | y = k) \cdot |\mathcal{B}|$$

The volume $|\mathcal{B}|$ appears in both numerator and denominator, canceling out: $$P(y = k | \mathbf{x}) = \lim_{|\mathcal{B}| \to 0} \frac{f(\mathbf{x} | y = k) \cdot |\mathcal{B}| \cdot P(y = k)}{\sum_{j} f(\mathbf{x} | y = j) \cdot |\mathcal{B}| \cdot P(y = j)}$$

$$= \frac{f(\mathbf{x} | y = k) P(y = k)}{\sum_{j} f(\mathbf{x} | y = j) P(y = j)}$$

This is a proper posterior probability, summing to 1 across all classes.

While any continuous density function could theoretically be used in the Bayesian framework, the Gaussian (normal) distribution is the canonical choice for continuous features. This preference is not arbitrary—it rests on deep theoretical and practical foundations.

The Central Limit Theorem

Perhaps the most compelling argument comes from the Central Limit Theorem (CLT):

The sum (or average) of many independent random variables tends toward a normal distribution, regardless of the original distribution.

Many real-world measurements arise as the aggregate of many small, independent factors:

• Human height: sum of genetic and environmental factors
• Measurement errors: sum of many small instrument imperfections
• Financial returns: aggregate of many individual trades
• Temperature: result of countless molecular interactions

Consequence: When we don't know the true distribution, Gaussian is often a reasonable assumption for measurements that result from aggregated effects.

Maximum Entropy Principle

Among all continuous distributions with a given mean and variance, the Gaussian distribution has maximum entropy. In information-theoretic terms:

$$H[X] = -\int f(x) \log f(x) dx$$

Maximizing entropy subject to constraints on mean and variance yields the Gaussian. This means:

• If we only know the mean and variance, Gaussian assumes minimal additional structure
• It's the "least presumptuous" distribution given limited information
• Any other choice would impose additional assumptions not warranted by the data

Computational Tractability

Gaussian distributions have remarkable mathematical properties:

1. Closed-form density: Easy to evaluate $$f(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

2. Simple parameter estimation: MLE gives $\hat{\mu} = \bar{x}$ and $\hat{\sigma}^2 = \frac{1}{n}\sum_i(x_i - \bar{x})^2$

3. Log-density is quadratic: Enables linear algebra in log-space $$\log f(x | \mu, \sigma^2) = -\frac{(x-\mu)^2}{2\sigma^2} - \frac{1}{2}\log(2\pi\sigma^2)$$

4. Closure under linear operations: Sums of Gaussians are Gaussian

5. Conjugate prior structure: Enables Bayesian inference

Practical Performance

Empirical evidence consistently shows that Gaussian Naive Bayes:

• Performs well on continuous data even when the true distribution is non-Gaussian
• Is robust to moderate violations of the Gaussian assumption
• Often matches or exceeds more complex methods on classification accuracy
• Provides well-calibrated probability estimates for many applications

Having established that we can use probability densities in Bayes' theorem and that Gaussians are a principled choice, we now apply the naive Bayes assumption to factorize the joint class-conditional density.

The Conditional Independence Assumption

For a feature vector $\mathbf{x} = (x_1, x_2, \ldots, x_d)$, the naive Bayes assumption states: $$f(\mathbf{x} | y = k) = \prod_{j=1}^{d} f(x_j | y = k)$$

Features are assumed to be conditionally independent given the class label. This allows us to model each feature's distribution separately within each class.

Gaussian Naive Bayes Model

Under Gaussian Naive Bayes, each univariate conditional density is Gaussian: $$f(x_j | y = k) = \mathcal{N}(x_j | \mu_{jk}, \sigma_{jk}^2) = \frac{1}{\sqrt{2\pi\sigma_{jk}^2}} \exp\left(-\frac{(x_j - \mu_{jk})^2}{2\sigma_{jk}^2}\right)$$

Where:

• $\mu_{jk}$ is the mean of feature $j$ for class $k$
• $\sigma_{jk}^2$ is the variance of feature $j$ for class $k$

The Complete Model

The full Gaussian Naive Bayes classifier is: $$f(\mathbf{x} | y = k) = \prod_{j=1}^{d} \frac{1}{\sqrt{2\pi\sigma_{jk}^2}} \exp\left(-\frac{(x_j - \mu_{jk})^2}{2\sigma_{jk}^2}\right)$$

Taking the logarithm (for numerical stability and convenience): $$\log f(\mathbf{x} | y = k) = -\frac{1}{2} \sum_{j=1}^{d} \left[ \frac{(x_j - \mu_{jk})^2}{\sigma_{jk}^2} + \log(2\pi\sigma_{jk}^2) \right]$$

Classification Rule

To classify a new point $\mathbf{x}$, compute: $$\hat{y} = \arg\max_k \left[ \log P(y = k) + \sum_{j=1}^{d} \log f(x_j | y = k) \right]$$

$$= \arg\max_k \left[ \log \pi_k - \frac{1}{2} \sum_{j=1}^{d} \left( \frac{(x_j - \mu_{jk})^2}{\sigma_{jk}^2} + \log \sigma_{jk}^2 \right) \right]$$

Where $\pi_k = P(y = k)$ is the class prior.

Understanding the number of parameters in Gaussian Naive Bayes reveals why it's such an efficient model and why the naive independence assumption is so powerful.

Counting Parameters

For a problem with $K$ classes, $d$ features:

Class priors: $K - 1$ free parameters (the $K$-th is determined by constraint $\sum_k \pi_k = 1$)

Gaussian parameters:

• Each feature in each class has a mean $\mu_{jk}$: $K \times d$ means
• Each feature in each class has a variance $\sigma_{jk}^2$: $K \times d$ variances

Total parameters: $$\text{Parameters} = (K - 1) + 2Kd = 2Kd + K - 1$$

For typical cases:

• Binary classification ($K = 2$), 10 features: $2 \times 2 \times 10 + 1 = 41$ parameters
• 10 classes, 100 features: $2 \times 10 \times 100 + 9 = 2009$ parameters

Comparison: Full Covariance Model

Without the naive Bayes assumption, we would model the full class-conditional distribution as a multivariate Gaussian: $$f(\mathbf{x} | y = k) = \mathcal{N}(\mathbf{x} | \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k)$$

A full $d \times d$ covariance matrix has $\frac{d(d+1)}{2}$ parameters per class: $$\text{Full model parameters} = Kd + K \cdot \frac{d(d+1)}{2} = Kd + \frac{Kd(d+1)}{2}$$

For 10 classes, 100 features: $10 \times 100 + 10 \times \frac{100 \times 101}{2} = 1000 + 50,500 = 51,500$ parameters!

The Diagonal Covariance Interpretation

The naive Bayes assumption is equivalent to assuming a diagonal covariance matrix:

$$\boldsymbol{\Sigma}k = \begin{pmatrix} \sigma{1k}^2 & 0 & \cdots & 0 \ 0 & \sigma_{2k}^2 & \cdots & 0 \ \vdots & \vdots & \ddots & \vdots \ 0 & 0 & \cdots & \sigma_{dk}^2 \end{pmatrix}$$

Zero off-diagonal entries mean zero correlation between features (conditional on class). This is exactly the conditional independence assumption expressed in linear algebra.

Key insight: Gaussian Naive Bayes assumes that knowing feature $x_1$ tells us nothing about feature $x_2$, once we know the class. The features may still be marginally correlated (in the overall data), but within each class, they are assumed independent.

Sample Complexity Implications

With fewer parameters, Gaussian Naive Bayes can be trained reliably with less data:

• Naive Bayes: Needs enough samples to estimate $2d$ parameters per class (means and variances)
• Full covariance: Needs enough samples to estimate $\frac{d(d+1)}{2}$ covariance parameters per class

Rule of thumb: Need at least 5-10× as many samples as parameters for reliable estimation. For 100 features:

• Naive Bayes: $\sim$1,000-2,000 samples per class
• Full covariance: $\sim$25,000-50,000 samples per class

This explains why Naive Bayes often outperforms more sophisticated models on small-to-medium datasets.

We have established the theoretical and practical foundations for applying Naive Bayes to continuous features. Let us consolidate the key concepts:

What's next:

Having understood why and how we model continuous features with Gaussians, the next page dives deep into the Gaussian distribution itself. We will explore its mathematical properties, the role of mean and variance parameters, and develop intuition for how Gaussian shapes encode information about data distributions.