Generalized Linear Models (GLMs)

Throughout your journey in machine learning, you've encountered a fundamental tension: linear regression assumes a continuous, unbounded response variable with normally distributed errors—yet real-world data often violates these assumptions dramatically. What do you do when your response is a count of events? A probability? A strictly positive amount? A categorical outcome?

For decades, statisticians developed specialized techniques for each case: logistic regression for binary outcomes, Poisson regression for counts, gamma regression for positive continuous values. These methods seemed unrelated—each with its own derivation, assumptions, and estimation procedures.

Then came the Generalized Linear Model (GLM) framework, introduced by John Nelder and Robert Wedderburn in their landmark 1972 paper. GLMs revealed that these seemingly disparate methods are special cases of a single, elegant mathematical structure. This unification wasn't merely taxonomic—it provided deep insights into when and why each method works, how to extend them, and how to diagnose their failures.

Before introducing the GLM framework, we must clearly understand why ordinary linear regression (OLS) fails for many real-world problems. This understanding motivates the need for a more general approach.

The standard linear regression model assumes:

$$Y_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \cdots + \beta_p X_{ip} + \varepsilon_i$$

where the errors $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$ are independent and identically distributed (iid) normal random variables. Equivalently:

$$Y_i \sim \mathcal{N}(\mu_i, \sigma^2), \quad \mu_i = \mathbf{x}_i^\top \boldsymbol{\beta}$$

This formulation embeds three critical assumptions that severely limit applicability:

Concrete examples of failure:

Example 1: Binary Classification Suppose you want to predict whether a patient has a disease (Y = 1) or not (Y = 0) based on clinical features. Fitting linear regression would produce predictions $\hat{Y}$ that can be negative or greater than 1—nonsensical as probabilities. Moreover, the variance of Y when Y ∈ {0,1} is p(1-p), which depends on the probability p, violating homoscedasticity.

Example 2: Count Data You want to model the number of customer complaints per day. Complaints cannot be negative, and the variance often increases with the expected count (more complaints means more variability). Linear regression might predict -3 complaints on a slow day.

Example 3: Positive Continuous Data You want to predict insurance claim amounts, which must be strictly positive and often have right-skewed distributions. Linear regression could predict negative claim amounts and assumes symmetric errors.

A Generalized Linear Model consists of three essential components that work together to model the relationship between predictors and response. Understanding these components is the key to mastering the entire framework.

Component 1: The Random Component (Distribution)

The random component specifies the probability distribution of the response variable $Y_i$ conditional on the predictors. In a GLM, this distribution must belong to the exponential family—a rich class of distributions including normal, binomial, Poisson, gamma, inverse Gaussian, and many others.

Formally, we specify that:

$$Y_i \sim \text{ExponentialFamily}(\theta_i, \phi)$$

where $\theta_i$ is the canonical (natural) parameter that depends on the predictors, and $\phi$ is a dispersion parameter that controls the variance.

The choice of distribution should reflect your prior knowledge about the data-generating process:

• Binary outcomes → Bernoulli/Binomial
• Counts without an upper bound → Poisson
• Positive continuous values → Gamma or Inverse Gaussian
• Continuous, unbounded values → Normal

Component 2: The Systematic Component (Linear Predictor)

The systematic component is the linear predictor $\eta_i$, which captures how predictors combine to influence the response:

$$\eta_i = \beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \cdots + \beta_p X_{ip} = \mathbf{x}_i^\top \boldsymbol{\beta}$$

This is the only place where predictors appear in the model. The linear predictor is always a linear combination of parameters $\boldsymbol{\beta}$, though the predictors $X_{ij}$ themselves can be nonlinear transformations of raw features (polynomials, interactions, basis expansions).

Critical insight: The systematic component is identical across all GLMs. Whether you're fitting logistic regression, Poisson regression, or any other GLM, the linear predictor has the same form. What differs is how this linear predictor relates to the response distribution.

Component 3: The Link Function

The link function $g(\cdot)$ is the critical connection between the random and systematic components. It relates the expected value of Y to the linear predictor:

$$g(\mu_i) = \eta_i = \mathbf{x}_i^\top \boldsymbol{\beta}$$

where $\mu_i = E[Y_i | \mathbf{x}_i]$ is the conditional mean of the response.

Equivalently, the mean is expressed as the inverse of the link function applied to the linear predictor:

$$\mu_i = g^{-1}(\eta_i) = g^{-1}(\mathbf{x}_i^\top \boldsymbol{\beta})$$

The link function serves two essential purposes:

1. Range Matching: It transforms the mean, which may be constrained (e.g., between 0 and 1), to the real line where the linear predictor lives
2. Interpretation: It determines how a unit change in a predictor affects the response—additively, multiplicatively, or otherwise

Now that we've introduced the three components conceptually, let's formalize the complete GLM specification. This mathematical precision is essential for understanding estimation, inference, and the relationships between different models.

Complete GLM Specification:

For independent observations $(Y_i, \mathbf{x}_i)$, $i = 1, \ldots, n$, a Generalized Linear Model is defined by:

1. Random Component: $$Y_i \mid \mathbf{x}_i \sim F(\mu_i, \phi)$$

where $F$ is from the exponential family with mean $\mu_i$ and dispersion $\phi$.

2. Systematic Component: $$\eta_i = \mathbf{x}i^\top \boldsymbol{\beta} = \sum{j=0}^{p} \beta_j x_{ij}$$

3. Link Function: $$g(\mu_i) = \eta_i$$

The link function $g: \mathcal{M} \to \mathbb{R}$ is a smooth, monotonic function that maps the mean space $\mathcal{M}$ (which may be restricted, like $(0,1)$ or $(0, \infty)$) to the entire real line.

The Mean-Variance Relationship:

For exponential family distributions, the variance is a function of the mean:

$$\text{Var}(Y_i) = \phi \cdot V(\mu_i)$$

where $V(\mu)$ is the variance function characteristic of the distribution. This relationship is fundamental:

This variance function encodes a key property: the variance changes with the mean in a distribution-specific way. GLMs automatically account for this heteroscedasticity, unlike OLS.

The power of the GLM framework becomes evident when we see how familiar models emerge as special cases. Each choice of distribution and link function yields a different member of the GLM family.

Linear Regression as a GLM

Distribution: $Y_i \sim \mathcal{N}(\mu_i, \sigma^2)$ (Normal) Link Function: $g(\mu) = \mu$ (Identity link) Inverse Link: $\mu = \eta$ Variance Function: $V(\mu) = 1$

$$\mu_i = \mathbf{x}_i^\top \boldsymbol{\beta}$$

With the identity link and normal distribution, GLM reduces exactly to ordinary least squares. The mean equals the linear predictor, and variance is constant.

Logistic Regression as a GLM

Distribution: $Y_i \sim \text{Bernoulli}(p_i)$ (Binomial with n=1) Link Function: $g(p) = \log\left(\frac{p}{1-p}\right)$ (Logit link) Inverse Link: $p = \frac{e^\eta}{1 + e^\eta} = \frac{1}{1 + e^{-\eta}}$ Variance Function: $V(p) = p(1-p)$

$$\log\left(\frac{p_i}{1-p_i}\right) = \mathbf{x}_i^\top \boldsymbol{\beta}$$

The logit link maps probabilities in (0,1) to the real line. The log-odds (logit) has a linear relationship with predictors. The variance is maximal at p=0.5 and decreases toward 0 and 1.

Poisson Regression as a GLM

Distribution: $Y_i \sim \text{Poisson}(\lambda_i)$ Link Function: $g(\mu) = \log(\mu)$ (Log link) Inverse Link: $\mu = e^\eta$ Variance Function: $V(\mu) = \mu$

$$\log(\lambda_i) = \mathbf{x}_i^\top \boldsymbol{\beta}$$

The log link ensures the mean count is always positive ($e^\eta > 0$ for any $\eta$). Predictors have multiplicative effects on the mean: a unit increase in $X_j$ multiplies the expected count by $e^{\beta_j}$.

Gamma Regression as a GLM

Distribution: $Y_i \sim \text{Gamma}(\alpha, \beta_i)$ (parameterized with mean $\mu_i$) Link Function: $g(\mu) = -1/\mu$ (Inverse link) or $g(\mu) = \log(\mu)$ (Log link) Variance Function: $V(\mu) = \mu^2$

Gamma regression is ideal for positive continuous data where variance increases with the square of the mean—common in financial data, insurance claims, and survival times.

For each exponential family distribution, there exists a special link function called the canonical link. This link has deep mathematical significance and practical advantages.

Definition of the Canonical Link

Recall that the density of an exponential family distribution can be written as:

$$f(y; \theta, \phi) = \exp\left{\frac{y\theta - b(\theta)}{a(\phi)} + c(y, \phi)\right}$$

where:

• $\theta$ is the canonical (natural) parameter
• $\phi$ is the dispersion parameter
• $b(\theta)$ is the cumulant function (determines the mean and variance)

The mean is related to the canonical parameter by: $$\mu = b'(\theta) \quad \Rightarrow \quad \theta = (b')^{-1}(\mu)$$

The canonical link is defined as: $$g(\mu) = \theta = (b')^{-1}(\mu)$$

In words: the canonical link is the function that maps the mean to the canonical parameter.

One of the most important skills in applied GLM modeling is correctly interpreting the estimated parameters. The interpretation depends critically on the link function used.

The General Principle

In a GLM with link function $g$:

$$g(\mu) = \beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p$$

The coefficient $\beta_j$ represents the change in $g(\mu)$ associated with a one-unit increase in $X_j$, holding all other predictors constant.

To interpret in terms of the mean $\mu$, we must apply $g^{-1}$—which gives different interpretations for different links.

Having established the mathematical structure of GLMs, let's step back and understand why this unified framework is so important for practical machine learning and statistics.

Connection to Modern Machine Learning:

GLMs might seem like classical statistics, but they form the backbone of modern machine learning:

1. Neural Network Output Layers: The final layer of a neural network for classification or regression is essentially a GLM—the softmax for multi-class classification is a multinomial logit GLM; the linear output for regression is a Gaussian GLM.

2. Gradient Boosting: XGBoost, LightGBM, and CatBoost can all be configured with different loss functions that correspond to different GLM distributions.

3. Bayesian Methods: GLMs have natural Bayesian extensions with conjugate priors, forming the foundation for modern probabilistic programming.

4. Transfer Learning: Understanding GLMs helps you design appropriate loss functions and output transformations when adapting pretrained models to new tasks.

We've covered the foundational architecture of Generalized Linear Models. Let's consolidate the key concepts:

What's Next:

In the next page, we'll dive deep into link functions—the critical component that connects the distribution mean to the linear predictor. We'll explore the properties of common link functions, understand how to choose between them, and see how the choice of link affects model behavior and interpretation.