Estimation Theory

At the heart of machine learning lies a deceptively simple question: Given observed data, how do we determine the parameters of the underlying model that generated it? This is the estimation problem, and its solution forms the theoretical backbone of nearly every learning algorithm you'll ever encounter.

Imagine you flip a coin 100 times and observe 67 heads. What's the probability of heads for this coin? Intuitively, you might guess 0.67—but can you justify this mathematically? Why not 0.5 (the fair coin assumption)? Why not 0.65 or 0.70? Maximum Likelihood Estimation (MLE) provides the rigorous framework to answer this question and, more importantly, generalizes to arbitrarily complex models.

Before diving into mathematics, we must understand what we're trying to achieve with parameter estimation. This philosophical grounding will guide our technical development and help us recognize when different methods are appropriate.

The fundamental assumption:

We assume our data comes from some underlying probability distribution characterized by unknown parameters. Our goal is to infer these parameters from observations. This is an inverse problem—we observe outputs (data) and want to determine inputs (parameters).

Consider these concrete scenarios:

Two competing philosophies:

There are two fundamental approaches to parameter estimation:

1. Frequentist approach: Parameters are fixed (but unknown) constants. We seek a point estimate—a single "best" value that somehow optimizes our knowledge given the data. MLE falls in this camp.

2. Bayesian approach: Parameters are random variables with their own probability distributions. We maintain full distributions over parameters, updating our beliefs as data arrives. Maximum A Posteriori (MAP) estimation bridges these views.

This page focuses on the frequentist MLE approach. We'll explore MAP in the next page, where the complementary perspectives become clear.

The likelihood function is the mathematical object at the heart of MLE. Its definition is deceptively simple but has profound implications.

Definition:

Given observed data D = {x₁, x₂, ..., xₙ} and a parametric model with parameters θ, the likelihood function is:

$$\mathcal{L}(\theta | \mathbf{D}) = P(\mathbf{D} | \theta)$$

This reads as: "the likelihood of parameters θ given data D equals the probability of observing data D given parameters θ."

Critical distinction: Probability vs. Likelihood

This is where many learners stumble. The mathematical formula is the same, but the interpretation differs:

The i.i.d. assumption:

When data points are independent and identically distributed (i.i.d.), the joint probability factorizes into a product:

$$\mathcal{L}(\theta | \mathbf{D}) = P(x_1, x_2, ..., x_n | \theta) = \prod_{i=1}^{n} P(x_i | \theta)$$

This is the key computational enabler. Without independence, we'd need to model complex dependencies between all data points—often intractable. The i.i.d. assumption turns an n-dimensional problem into n one-dimensional problems multiplied together.

Example: Bernoulli likelihood

Consider n coin flips with unknown probability p of heads. If we observe k heads and (n-k) tails:

$$\mathcal{L}(p | k, n) = \binom{n}{k} p^k (1-p)^{n-k}$$

For p = 0.5, 0.6, 0.7 with n = 100, k = 67:

• L(0.5) = C(100,67) × 0.5¹⁰⁰ ≈ 1.4 × 10⁻⁶
• L(0.6) = C(100,67) × 0.6⁶⁷ × 0.4³³ ≈ 4.1 × 10⁻³
• L(0.7) = C(100,67) × 0.7⁶⁷ × 0.3³³ ≈ 8.6 × 10⁻³

The likelihood at p = 0.7 is vastly higher than at p = 0.5—quantifying our intuition that p ≈ 0.67 is more plausible than p = 0.5.

With the likelihood function defined, the MLE principle is elegantly simple:

MLE Principle:

Choose the parameter value θ that maximizes the likelihood of the observed data:

$$\hat{\theta}{MLE} = \arg\max{\theta} \mathcal{L}(\theta | \mathbf{D}) = \arg\max_{\theta} \prod_{i=1}^{n} P(x_i | \theta)$$

The hat notation (θ̂) denotes an estimate—our best guess based on data, as opposed to the true (unknowable) parameter θ*.

Intuition:

If we must pick a single value for θ, we should pick the one that makes the observed data most probable. Any other choice would imply we think the data we actually saw was less likely than data we didn't see—a philosophically awkward position.

The optimization challenge:

Finding the maximum of L(θ) requires calculus. We seek:

$$\frac{\partial \mathcal{L}(\theta)}{\partial \theta} = 0$$

However, differentiating products is messy. This motivates the log-likelihood transformation.

The Log-Likelihood Transformation:

The logarithm is a monotonically increasing function: if a > b, then log(a) > log(b). This means maximizing L(θ) is equivalent to maximizing log L(θ):

$$\hat{\theta}{MLE} = \arg\max{\theta} \mathcal{L}(\theta) = \arg\max_{\theta} \ell(\theta)$$

where ℓ(θ) = log L(θ) is the log-likelihood.

Why log-likelihood is computationally superior:

Let's derive the MLE for the simplest non-trivial case: estimating the bias of a coin. This example illustrates the complete MLE workflow.

Problem setup:

• Data: n independent coin flips D = {x₁, x₂, ..., xₙ} where xᵢ ∈ {0, 1}
• Model: Bernoulli(p) with unknown p ∈ [0, 1]
• Goal: Find p̂ₘₗₑ that maximizes the likelihood

Step 1: Write the likelihood function

$$\mathcal{L}(p | \mathbf{D}) = \prod_{i=1}^{n} P(x_i | p) = \prod_{i=1}^{n} p^{x_i}(1-p)^{1-x_i}$$

Simplifying (let k = Σxᵢ = number of heads):

$$\mathcal{L}(p) = p^k (1-p)^{n-k}$$

Step 2: Transform to log-likelihood

$$\ell(p) = \log \mathcal{L}(p) = k \log p + (n-k) \log(1-p)$$

Step 3: Differentiate and set to zero

$$\frac{d\ell}{dp} = \frac{k}{p} - \frac{n-k}{1-p} = 0$$

Step 4: Solve for p

Multiplying through: $$k(1-p) = (n-k)p$$ $$k - kp = np - kp$$ $$k = np$$ $$\boxed{\hat{p}_{MLE} = \frac{k}{n}}$$

The MLE for a Bernoulli parameter is simply the sample proportion!

This matches intuition perfectly: if you flip a coin 100 times and see 67 heads, your best estimate for P(heads) is 67/100 = 0.67.

Step 5: Verify it's a maximum (not minimum)

We should check the second derivative:

$$\frac{d^2\ell}{dp^2} = -\frac{k}{p^2} - \frac{n-k}{(1-p)^2}$$

For p ∈ (0, 1) with k > 0 and k < n, this is always negative, confirming we have a maximum.

Edge cases:

• If k = 0 (all tails): p̂ = 0
• If k = n (all heads): p̂ = 1

These extreme estimates illustrate a weakness of MLE with limited data—there's no regularization pulling the estimate toward reasonable values. MAP estimation addresses this.

The Gaussian (Normal) distribution is perhaps the most important in all of statistics and machine learning. Let's derive the MLE for both parameters: mean μ and variance σ².

Problem setup:

• Data: n i.i.d. observations D = {x₁, x₂, ..., xₙ}
• Model: N(μ, σ²) with unknown μ and σ²
• Goal: Find (μ̂, σ̂²)ₘₗₑ

Step 1: Write the likelihood function

The Gaussian PDF is:

$$P(x | \mu, \sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$$

For n i.i.d. observations:

$$\mathcal{L}(\mu, \sigma^2) = \prod_{i=1}^{n} \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x_i-\mu)^2}{2\sigma^2}\right)$$

$$= \left(\frac{1}{2\pi\sigma^2}\right)^{n/2} \exp\left(-\frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2\right)$$

Step 2: Transform to log-likelihood

$$\ell(\mu, \sigma^2) = -\frac{n}{2}\log(2\pi) - \frac{n}{2}\log(\sigma^2) - \frac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2$$

Step 3: Differentiate with respect to μ

$$\frac{\partial \ell}{\partial \mu} = \frac{1}{\sigma^2}\sum_{i=1}^n(x_i - \mu) = 0$$

$$\sum_{i=1}^n x_i - n\mu = 0$$

$$\boxed{\hat{\mu}{MLE} = \frac{1}{n}\sum{i=1}^n x_i = \bar{x}}$$

The MLE for the Gaussian mean is the sample mean!

Step 4: Differentiate with respect to σ²

Let τ = σ² for clarity:

$$\frac{\partial \ell}{\partial \tau} = -\frac{n}{2\tau} + \frac{1}{2\tau^2}\sum_{i=1}^n(x_i-\mu)^2 = 0$$

Substituting μ̂ = x̄:

$$\frac{n}{2\tau} = \frac{1}{2\tau^2}\sum_{i=1}^n(x_i-\bar{x})^2$$

$$\boxed{\hat{\sigma}^2_{MLE} = \frac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2}$$

The MLE for variance is the sample variance with n in the denominator.

Important note on bias:

The MLE estimator σ̂²ₘₗₑ is biased! Its expected value is:

$$E[\hat{\sigma}^2_{MLE}] = \frac{n-1}{n}\sigma^2 eq \sigma^2$$

This is why the unbiased sample variance uses (n-1) in the denominator (Bessel's correction). We'll explore bias in depth on Page 3.

The Exponential distribution models waiting times and is fundamental in reliability analysis, queueing theory, and survival analysis.

Problem setup:

• Data: n i.i.d. waiting times D = {x₁, x₂, ..., xₙ} where xᵢ > 0
• Model: Exponential(λ) with rate parameter λ > 0
• Goal: Find λ̂ₘₗₑ

The Exponential PDF:

$$P(x | \lambda) = \lambda e^{-\lambda x}, \quad x > 0$$

The mean of this distribution is 1/λ.

Step 1: Write the log-likelihood

$$\ell(\lambda) = \sum_{i=1}^n \log(\lambda e^{-\lambda x_i}) = n\log\lambda - \lambda\sum_{i=1}^n x_i$$

Step 2: Differentiate and solve

$$\frac{d\ell}{d\lambda} = \frac{n}{\lambda} - \sum_{i=1}^n x_i = 0$$

$$\boxed{\hat{\lambda}{MLE} = \frac{n}{\sum{i=1}^n x_i} = \frac{1}{\bar{x}}}$$

The MLE for the rate is the reciprocal of the sample mean!

Intuition: If you observe average waiting times of 5 minutes, your best estimate for the rate is 1/5 = 0.2 events per minute.

Verify it's a maximum:

$$\frac{d^2\ell}{d\lambda^2} = -\frac{n}{\lambda^2} < 0 \text{ for all } \lambda > 0 \checkmark$$

The examples above—Bernoulli, Gaussian, Exponential—are relatively simple. But MLE's true power emerges when applied to complex models. Nearly every supervised learning algorithm can be understood as MLE under an appropriate probabilistic interpretation.

Linear Regression as MLE:

Consider the model y = w^T x + ε where ε ~ N(0, σ²). The likelihood of observing output y given input x and weights w is:

$$P(y | \mathbf{x}, \mathbf{w}) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y - \mathbf{w}^T\mathbf{x})^2}{2\sigma^2}\right)$$

The negative log-likelihood (NLL) for n observations:

$$-\ell(\mathbf{w}) = \frac{n}{2}\log(2\pi\sigma^2) + \frac{1}{2\sigma^2}\sum_{i=1}^n(y_i - \mathbf{w}^T\mathbf{x}_i)^2$$

Minimizing NLL is equivalent to minimizing Mean Squared Error! The OLS solution IS the MLE.

Logistic Regression as MLE:

For binary classification, we model P(y=1|x,w) = σ(w^T x) where σ is the sigmoid function.

The likelihood for one observation: $$P(y | \mathbf{x}, \mathbf{w}) = \sigma(\mathbf{w}^T\mathbf{x})^y (1 - \sigma(\mathbf{w}^T\mathbf{x}))^{1-y}$$

The negative log-likelihood is the Binary Cross-Entropy Loss: $$-\ell(\mathbf{w}) = -\sum_{i=1}^n [y_i \log \sigma(\mathbf{w}^T\mathbf{x}_i) + (1-y_i) \log(1 - \sigma(\mathbf{w}^T\mathbf{x}_i))]$$

Neural Networks as MLE:

Deep learning with cross-entropy loss is MLE under a multinomial (softmax) output distribution. The network learns to maximize the likelihood of correct class probabilities.

MLE isn't just convenient—it has remarkable theoretical properties that make it the method of choice for many estimation problems. These properties explain why MLE dominates practical machine learning.

Key Asymptotic Properties (as n → ∞):

The Fisher Information:

The Fisher Information measures how much the likelihood function curves around the true parameter—essentially, how much information the data provides about θ:

$$I(\theta) = -E\left[\frac{\partial^2 \log P(X|\theta)}{\partial \theta^2}\right] = E\left[\left(\frac{\partial \log P(X|\theta)}{\partial \theta}\right)^2\right]$$

High Fisher Information → Sharply peaked likelihood → Precise estimates possible Low Fisher Information → Flat likelihood → Estimation is difficult

The Cramér-Rao Lower Bound:

For any unbiased estimator θ̂:

$$\text{Var}(\hat{\theta}) \geq \frac{1}{nI(\theta)}$$

MLE asymptotically achieves this bound—it extracts the maximum possible information from data.

Moving from theory to practice requires attention to several computational and numerical details:

1. Numerical Optimization:

For complex models without closed-form solutions, we use iterative optimization:

• Gradient Descent: θ ← θ - α∇ℓ(θ)
• Newton's Method: θ ← θ - H⁻¹∇ℓ(θ) where H is the Hessian
• Quasi-Newton (L-BFGS): Approximates Hessian efficiently
• Stochastic Gradient Descent: Essential for large datasets

2. Numerical Stability:

We've covered the foundational theory and practice of Maximum Likelihood Estimation. Let's consolidate the key insights:

Looking ahead:

MLE treats parameters as fixed unknowns and seeks point estimates. But what if we want to express uncertainty about parameters? What if we have prior beliefs we want to incorporate? The next page introduces Maximum A Posteriori (MAP) estimation, which bridges MLE with Bayesian thinking and provides the foundation for regularization in machine learning.