Common Probability Distributions

Every classification problem in machine learning—spam detection, disease diagnosis, fraud identification, sentiment analysis—fundamentally reduces to predicting binary outcomes. Will the email be spam or not? Is the tumor malignant or benign? Is this transaction fraudulent or legitimate?

To model these binary outcomes probabilistically, we need a mathematical framework that captures the inherent randomness of such yes/no decisions. This is precisely what the Bernoulli and Binomial distributions provide. They are the simplest yet most fundamental probability distributions, forming the conceptual bedrock upon which more sophisticated models—logistic regression, naive Bayes classifiers, and neural network classification heads—are built.

Understanding these distributions is not merely an academic exercise; it is essential for grasping how machine learning models quantify uncertainty, how we evaluate classifier performance, and how we make principled decisions under probabilistic predictions.

The Bernoulli distribution is the simplest discrete probability distribution, modeling a single trial with exactly two possible outcomes: success (typically encoded as 1) and failure (encoded as 0). Named after Swiss mathematician Jacob Bernoulli (1654–1705), who made foundational contributions to probability theory, this distribution captures the essence of random binary events.

Formal Definition

A random variable X follows a Bernoulli distribution with parameter p, written X ~ Bernoulli(p), if:

Probability Mass Function (PMF):

$$P(X = x) = p^x (1-p)^{1-x}, \quad x \in \{0, 1\}$$

Equivalently, we can write this as:

$$P(X = 1) = p \quad \text{and} \quad P(X = 0) = 1 - p = q$$

where:

• p ∈ [0, 1] is the probability of success
• q = 1 - p is the probability of failure
• X takes only two values: 0 or 1

The parameter p completely characterizes the distribution. Once you know p, you know everything about the randomness of a single binary trial.

Moments of the Bernoulli Distribution

The moments (expected value, variance, etc.) of the Bernoulli distribution are remarkably simple yet carry profound implications for machine learning.

Expected Value (Mean):

$$E[X] = \sum_{x \in \{0,1\}} x \cdot P(X=x) = 0 \cdot (1-p) + 1 \cdot p = p$$

The mean of a Bernoulli random variable equals the probability of success. This is intuitive: if you flip a biased coin with P(heads) = 0.7, the average outcome is 0.7.

Variance:

$$Var(X) = E[X^2] - (E[X])^2$$

Since X ∈ {0, 1}, we have X² = X, so E[X²] = E[X] = p. Therefore:

$$Var(X) = p - p^2 = p(1-p) = pq$$

Standard Deviation:

$$\sigma = \sqrt{p(1-p)}$$

Key Insight on Variance: The variance pq is maximized when p = 0.5 (giving Var = 0.25) and minimized when p = 0 or p = 1 (giving Var = 0). This reflects the intuition that a fair coin (p = 0.5) has the most uncertainty, while a deterministic outcome (p = 0 or 1) has no uncertainty.

Higher Moments and Moment Generating Function

For deeper analysis, we can compute the moment generating function (MGF):

$$M_X(t) = E[e^{tX}] = (1-p) \cdot e^{0 \cdot t} + p \cdot e^{1 \cdot t} = (1-p) + pe^t$$

From the MGF, we can derive all moments by differentiation:

$$E[X^n] = \frac{d^n}{dt^n} M_X(t) \Big|_{t=0}$$

The characteristic function (Fourier transform of the PMF) is:

$$\phi_X(t) = E[e^{itX}] = (1-p) + pe^{it}$$

These functions are essential for proving limit theorems and analyzing sums of random variables.

While the Bernoulli distribution models a single binary trial, the Binomial distribution extends this to model the total number of successes in n independent, identical Bernoulli trials. This is the natural progression: from one coin flip to many flips, from one classification to many classifications.

Formal Definition

A random variable X follows a Binomial distribution with parameters n and p, written X ~ Binomial(n, p), if X represents the count of successes in n independent Bernoulli(p) trials.

Probability Mass Function (PMF):

$$P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}, \quad k \in \{0, 1, 2, \ldots, n\}$$

where the binomial coefficient is:

$$\binom{n}{k} = \frac{n!}{k!(n-k)!}$$

This coefficient counts the number of ways to choose which k trials are successes among n trials.

Derivation from First Principles

Why does the binomial PMF take this form? Consider n = 3 trials with P(success) = p.

For exactly k = 2 successes, we could have the patterns:

• SSF (success, success, failure): probability p · p · (1-p) = p²(1-p)
• SFS: probability p · (1-p) · p = p²(1-p)
• FSS: probability (1-p) · p · p = p²(1-p)

Each pattern has probability p²(1-p), and there are C(3,2) = 3 such patterns.

Total probability: P(X = 2) = 3 · p²(1-p)

Generalizing: P(X = k) = (number of ways to arrange k successes) × (probability of any specific arrangement)

$$P(X = k) = \binom{n}{k} \cdot p^k (1-p)^{n-k}$$

Moments of the Binomial Distribution

Since X = X₁ + X₂ + ... + Xₙ where each Xᵢ ~ Bernoulli(p), we can derive moments from linearity of expectation.

Expected Value:

$$E[X] = E\left[\sum_{i=1}^n X_i\right] = \sum_{i=1}^n E[X_i] = \sum_{i=1}^n p = np$$

Variance:

Since the Xᵢ are independent:

$$Var(X) = Var\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n Var(X_i) = \sum_{i=1}^n p(1-p) = np(1-p) = npq$$

Standard Deviation:

$$\sigma = \sqrt{np(1-p)}$$

Mode (most likely value):

• If (n+1)p is an integer, there are two modes: (n+1)p and (n+1)p - 1
• Otherwise, the mode is ⌊(n+1)p⌋

Moment Generating Function

The MGF of the binomial is derived from the MGF of the Bernoulli:

$$M_X(t) = E[e^{tX}] = E\left[e^{t(X_1 + \cdots + X_n)}\right] = \prod_{i=1}^n E[e^{tX_i}] = ((1-p) + pe^t)^n$$

This elegant form—the n-th power of the Bernoulli MGF—reflects the sum structure of the binomial.

The Normalization Property

A valid probability distribution must sum to 1. For the binomial:

$$\sum_{k=0}^n \binom{n}{k} p^k (1-p)^{n-k} = (p + (1-p))^n = 1^n = 1 \quad \checkmark$$

This follows directly from the binomial theorem, which is why the distribution bears this name.

Understanding probability distributions through visualization deepens intuition beyond formulas. Let's examine how Bernoulli and Binomial distributions behave under different parameter settings.

Bernoulli PMF Visualization

The Bernoulli PMF is strikingly simple: two bars at x = 0 and x = 1, with heights (1-p) and p respectively.

Binomial PMF: Effect of n and p

The binomial PMF becomes richer as n increases, displaying a bell-shaped curve for moderate p values.

Key Visual Insights

Skewness behavior:

• When p < 0.5: Distribution is right-skewed (long tail toward high values)
• When p = 0.5: Distribution is symmetric
• When p > 0.5: Distribution is left-skewed (long tail toward low values)

Concentration around the mean: As n increases, the distribution becomes more concentrated around np (relative to the range). The coefficient of variation CV = σ/μ = √((1-p)/(np)) decreases as 1/√n, meaning the distribution becomes relatively tighter.

Approaching normality: For large n (typically np ≥ 5 and n(1-p) ≥ 5), the binomial approaches a normal distribution with mean np and variance np(1-p). This is the celebrated Central Limit Theorem in action.

In machine learning, we rarely know the true parameter p; instead, we must estimate it from observed data. This section develops the theory of parameter estimation for Bernoulli and Binomial distributions using Maximum Likelihood Estimation (MLE).

MLE for Bernoulli Parameter

Suppose we observe n independent samples x₁, x₂, ..., xₙ from a Bernoulli(p) distribution. The likelihood function is the probability of observing this specific data as a function of p:

$$L(p | x_1, \ldots, x_n) = \prod_{i=1}^n p^{x_i} (1-p)^{1-x_i} = p^{\sum x_i} (1-p)^{n - \sum x_i}$$

Let k = Σxᵢ be the total number of successes. Then:

$$L(p) = p^k (1-p)^{n-k}$$

The Log-Likelihood

Maximizing the likelihood is equivalent to maximizing its logarithm (since log is monotonic):

$$\ell(p) = \log L(p) = k \log p + (n-k) \log(1-p)$$

To find the maximum, we take the derivative and set it to zero:

$$\frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0$$

Solving for p:

$$\frac{k}{p} = \frac{n-k}{1-p}$$ $$k(1-p) = (n-k)p$$ $$k - kp = np - kp$$ $$k = np$$ $$\hat{p}{MLE} = \frac{k}{n} = \frac{\sum{i=1}^n x_i}{n}$$

The MLE estimate is simply the sample proportion—the number of successes divided by total trials.

Properties of the MLE Estimator

Unbiasedness: $$E[\hat{p}] = E\left[\frac{\sum X_i}{n}\right] = \frac{1}{n} \sum E[X_i] = \frac{1}{n} \cdot np = p$$

The estimator is unbiased—on average, it equals the true parameter.

Variance: $$Var(\hat{p}) = Var\left(\frac{\sum X_i}{n}\right) = \frac{1}{n^2} Var\left(\sum X_i\right) = \frac{1}{n^2} \cdot np(1-p) = \frac{p(1-p)}{n}$$

The variance decreases as 1/n—more data means more precision.

Standard Error: $$SE(\hat{p}) = \sqrt{\frac{p(1-p)}{n}}$$

Consistency: As n → ∞, the variance approaches 0, so p̂ → p in probability. The estimator is consistent.

Asymptotic Normality: For large n: $$\hat{p} \stackrel{d}{\rightarrow} N\left(p, \frac{p(1-p)}{n}\right)$$

This enables confidence intervals and hypothesis testing.

The Bernoulli and Binomial distributions are not isolated; they connect to a rich web of other probability distributions. Understanding these relationships deepens comprehension and reveals when different distributions are appropriate.

From Bernoulli to Binomial

The fundamental relationship: if X₁, X₂, ..., Xₙ are independent Bernoulli(p) random variables, then:

$$X = \sum_{i=1}^n X_i \sim \text{Binomial}(n, p)$$

Conversely, a Binomial(1, p) is simply a Bernoulli(p).

Binomial and Poisson (Poisson Limit Theorem)

When n is large, p is small, and λ = np is moderate, the binomial approximates the Poisson distribution:

$$\text{Binomial}(n, p) \approx \text{Poisson}(\lambda) \quad \text{where } \lambda = np$$

Precisely, as n → ∞ and p → 0 with np → λ:

$$\binom{n}{k} p^k (1-p)^{n-k} \rightarrow \frac{\lambda^k e^{-\lambda}}{k!}$$

Rule of thumb: Use Poisson when n ≥ 20 and p ≤ 0.05.

Example: If 1 in 1000 emails is spam (p = 0.001) and you receive 500 emails (n = 500), the number of spam emails approximates Poisson(λ = 0.5).

Normal Approximation (De Moivre-Laplace Theorem)

For large n with np ≥ 5 and n(1-p) ≥ 5:

$$\frac{X - np}{\sqrt{np(1-p)}} \stackrel{d}{\rightarrow} N(0, 1)$$

Or equivalently:

$$X \approx N(np, np(1-p))$$

Continuity Correction: Since we're approximating a discrete distribution with a continuous one, we improve accuracy with a continuity correction:

$$P(X \leq k) \approx \Phi\left(\frac{k + 0.5 - np}{\sqrt{np(1-p)}}\right)$$

Beta-Binomial Connection

In Bayesian inference, if we place a Beta(α, β) prior on the parameter p of a Binomial, the posterior after observing k successes in n trials is:

$$p | (k, n) \sim \text{Beta}(\alpha + k, \beta + n - k)$$

This conjugacy is fundamental to Bayesian statistics and leads to the Beta-Binomial compound distribution.

Bernoulli and Binomial distributions permeate machine learning, underpinning fundamental algorithms and evaluation methodologies. Let's explore their key applications in depth.

Binary Classification and Logistic Regression

Logistic regression models the probability of class membership using a Bernoulli distribution:

$$P(Y = 1 | \mathbf{x}) = \sigma(\mathbf{w}^T \mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^T \mathbf{x} + b)}}$$

Given the predicted probability p = σ(w^T x + b), the target Y follows:

$$Y | \mathbf{x} \sim \text{Bernoulli}(\sigma(\mathbf{w}^T \mathbf{x} + b))$$

The negative log-likelihood loss (binary cross-entropy) is:

$$\mathcal{L} = -\sum_{i=1}^n \left[ y_i \log(p_i) + (1-y_i) \log(1-p_i) \right]$$

This is precisely the negative log-likelihood of Bernoulli observations—training logistic regression is MLE for Bernoulli parameters!

Naive Bayes Classification

The Bernoulli Naive Bayes classifier models features as independent Bernoulli random variables given the class:

$$P(x_j = 1 | y = c) = \theta_{jc}$$

For binary features (e.g., word presence/absence in text):

$$P(\mathbf{x} | y = c) = \prod_{j=1}^d \theta_{jc}^{x_j} (1 - \theta_{jc})^{1-x_j}$$

Classification uses Bayes' rule:

$$P(y = c | \mathbf{x}) \propto P(y = c) \prod_{j=1}^d P(x_j | y = c)$$

A/B Testing and Hypothesis Testing

A/B testing compares conversion rates (Bernoulli parameters) between control and treatment groups:

• Control: n₁ users, k₁ conversions → p̂₁ = k₁/n₁
• Treatment: n₂ users, k₂ conversions → p̂₂ = k₂/n₂

Test statistic: $$Z = \frac{\hat{p}_2 - \hat{p}_1}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}}$$

where p̂ is the pooled proportion.

Sample size calculation for A/B tests: $$n \approx \frac{2(z_{\alpha/2} + z_\beta)^2 \cdot p(1-p)}{(\delta)^2}$$

where δ is the minimum detectable effect.

Evaluating Classifier Performance

When evaluating a classifier on n test samples, the number of correct predictions X follows a binomial distribution:

$$X \sim \text{Binomial}(n, p_{\text{accuracy}})$$

The observed accuracy is a/n where a is the number of correct predictions. A 95% confidence interval for true accuracy uses the binomial properties:

• Normal approximation: a/n ± 1.96√(p̂(1-p̂)/n)
• Clopper-Pearson (exact): Based on relationship to Beta distribution
• Wilson score: Better coverage for small n or extreme accuracies

This is crucial for comparing models: "Model A achieved 85% and Model B achieved 83%"—is this difference significant? The binomial framework provides the answer.

While the Bernoulli distribution is computationally trivial—just a coin flip—the binomial involves factorials that can overflow for large n. Understanding computational approaches is essential for practical implementation.

Computing Binomial Coefficients

Naive calculation of C(n,k) = n!/(k!(n-k)!) fails for large n due to integer overflow. Better approaches:

1. Iterative calculation (avoids factorial overflow): $$\binom{n}{k} = \prod_{i=0}^{k-1} \frac{n-i}{i+1}$$

2. Log-space computation: $$\log \binom{n}{k} = \sum_{i=0}^{k-1} [\log(n-i) - \log(i+1)]$$

or using the log-gamma function: $$\log \binom{n}{k} = \log \Gamma(n+1) - \log \Gamma(k+1) - \log \Gamma(n-k+1)$$

Generating Random Samples

Efficient sampling from binomial distributions is important for Monte Carlo methods.

Method 1: Sum of Bernoulli trials (simple but slow for large n) $$X = \sum_{i=1}^n \text{Bernoulli}(p)$$

Method 2: Inverse transform (for small n) Compute the CDF and use uniform random numbers.

Method 3: BTPE algorithm (for large n) The Binomial Triangle Parallelogram Exponential (BTPE) algorithm combines multiple techniques for efficiency across all parameter ranges. This is what NumPy and most libraries use.

Numerical Precision

When p is very small or very large, direct probability calculations can underflow. Always:

1. Work in log-space for likelihoods
2. Use scipy.special.logsumexp for summing probabilities
3. Use specialized functions like scipy.stats.binom which handle edge cases
4. For ML training, use numerically stable loss implementations (e.g., torch.nn.BCEWithLogitsLoss)

The Bernoulli and Binomial distributions form the foundation for modeling binary outcomes in machine learning. Let's consolidate the essential concepts:

What's Next:

The Bernoulli and Binomial handle discrete binary outcomes. In the next page, we explore the Gaussian (Normal) distribution—the cornerstone of continuous probability modeling. We'll see how it arises as the limit of binomials (Central Limit Theorem), why it's ubiquitous in nature and ML, and how it underpins regression, neural networks, and much more.