Multivariate Calculus

In your first encounter with calculus, you learned to analyze functions of a single variable: f(x). You computed derivatives, found critical points, and understood how a function changes as its single input varies. But in machine learning, we inhabit a fundamentally different world—one where models depend on hundreds, thousands, or even millions of variables simultaneously.

Consider a simple neural network for image classification. Each pixel in a 224×224 RGB image contributes 3 values, yielding 150,528 input dimensions. The network's weights might add another 25 million parameters. How does the loss function change when we nudge one of these millions of values? How do we navigate this incomprehensibly vast space to find optimal parameters?

The answer lies in multivariate calculus—the mathematical framework for analyzing functions of multiple variables. This isn't merely an extension of single-variable calculus; it's a fundamentally richer theory that enables the optimization machinery powering modern machine learning.

A multivariable function (also called a multivariate function or function of several variables) is a function whose domain consists of ordered tuples of real numbers rather than single real numbers. Formally:

Definition (Multivariable Function):

A function f: ℝⁿ → ℝ maps an n-dimensional input vector x = (x₁, x₂, ..., xₙ) to a single real number:

$$f(\mathbf{x}) = f(x_1, x_2, \ldots, x_n)$$

More generally, a function f: ℝⁿ → ℝᵐ maps n-dimensional inputs to m-dimensional outputs, but we'll focus primarily on scalar-valued functions (m = 1) since these represent the loss functions we optimize in machine learning.

The Conceptual Leap:

In single-variable calculus, the graph of f(x) is a curve in 2D. In two-variable calculus, the graph of f(x, y) is a surface in 3D. But what about f(x₁, x₂, ..., x₁₀₀)? We cannot visualize 101-dimensional space, yet the mathematics works identically. This abstraction is powerful: the tools we develop for two or three variables extend naturally to arbitrary dimensions.

Notation Conventions:

Throughout machine learning, you'll encounter several equivalent notations:

• Explicit variables: f(x₁, x₂, ..., xₙ)
• Vector notation: f(x) where x ∈ ℝⁿ
• Parameter notation: f(x; θ) to distinguish inputs from model parameters
• Loss notation: L(θ) or J(θ) for loss as a function of parameters

Understanding the domain and range of multivariable functions is crucial for reasoning about machine learning models and their optimization landscapes.

Domain of a Multivariable Function:

The domain of f: ℝⁿ → ℝ is the set of all input vectors x for which f(x) is defined. Unlike single-variable functions where the domain is a subset of the real line, multivariable domains are subsets of n-dimensional space.

Common Domain Types in ML:

Range and Level Sets:

The range of f: ℝⁿ → ℝ is the set of all output values the function attains. For loss functions, the range is typically [0, ∞) or ℝ, depending on whether the loss is non-negative.

Level Sets (Contours):

A level set of f is the set of all points where f takes a constant value:

$$L_c = {\mathbf{x} \in \mathbb{R}^n : f(\mathbf{x}) = c}$$

For a function of two variables, level sets are curves called contour lines. For three variables, they're surfaces. For n variables, they're (n-1)-dimensional hypersurfaces.

Geometric Intuition:

Imagine a topographic map showing elevation. Each contour line connects points of equal height. Similarly, for a loss function L(θ), level sets connect all parameter configurations with the same loss. The gradient at any point is perpendicular to the level set, pointing toward higher values—a fact that's fundamental to gradient descent.

Let's ground these abstractions with concrete examples from machine learning. Each example illustrates how real ML models are fundamentally multivariable functions.

Example 1: Linear Regression Prediction

A linear regression model with n features computes:

$$f(\mathbf{x}; \mathbf{w}, b) = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w}^\top \mathbf{x} + b$$

This function takes x ∈ ℝⁿ as input and w ∈ ℝⁿ, b ∈ ℝ as parameters, producing a scalar output. The function is linear in both inputs and parameters.

Example 2: Mean Squared Error Loss

Given m training examples {(xᵢ, yᵢ)}, the MSE loss as a function of weights:

$$L(\mathbf{w}, b) = \frac{1}{m} \sum_{i=1}^{m} \left( \mathbf{w}^\top \mathbf{x}_i + b - y_i \right)^2$$

This is a function of n+1 parameters, mapping ℝⁿ⁺¹ → ℝ. It's a quadratic function—specifically, a positive definite quadratic form (bowl-shaped with a unique minimum) when the data matrix has full column rank.

Example 3: Softmax Cross-Entropy Loss

For K-class classification with input z (logits):

$$\text{softmax}(\mathbf{z})k = \frac{e^{z_k}}{\sum{j=1}^{K} e^{z_j}}$$

$$L(\mathbf{z}; y) = -\log\left( \text{softmax}(\mathbf{z})y \right) = -z_y + \log\left( \sum{j=1}^{K} e^{z_j} \right)$$

This is a nonlinear, convex function of z ∈ ℝᴷ.

Example 4: Neural Network Prediction

A two-layer neural network:

$$f(\mathbf{x}; \mathbf{W}^{(1)}, \mathbf{b}^{(1)}, \mathbf{W}^{(2)}, \mathbf{b}^{(2)}) = \mathbf{W}^{(2)} \sigma(\mathbf{W}^{(1)} \mathbf{x} + \mathbf{b}^{(1)}) + \mathbf{b}^{(2)}$$

Here σ is a nonlinear activation (ReLU, sigmoid, etc.). The loss landscape as a function of all weight matrices and biases is highly non-convex, with many local minima and saddle points.

Developing geometric intuition for multivariable functions is essential for understanding optimization behavior in machine learning.

Graphs and Surfaces:

For a function f: ℝ² → ℝ, the graph is the set of all points (x, y, f(x, y)) ∈ ℝ³—a surface in three-dimensional space. For a quadratic function like MSE loss, this surface is a paraboloid (bowl shape). For more complex functions, the surface can have peaks, valleys, saddle points, and ridges.

Cross-Sections and Slices:

We can understand high-dimensional functions by examining lower-dimensional slices:

• Fixing all but one variable: f(x₁) = f(x₁, c₂, c₃, ..., cₙ) gives a 1D curve
• Fixing all but two variables: f(x₁, x₂) = f(x₁, x₂, c₃, ..., cₙ) gives a 2D surface

This technique is invaluable for loss landscape visualization: we often pick two random directions in parameter space and plot the loss along them.

Direction and Distance:

In ℝⁿ, a direction is specified by a unit vector u with ‖u‖ = 1. Moving from point x in direction u by distance t gives:

$$\mathbf{x} + t\mathbf{u}$$

The function value along this ray is g(t) = f(x + tu), a single-variable function. This reduction to one dimension is the conceptual foundation for directional derivatives.

Before we can discuss derivatives of multivariable functions, we need to understand continuity in higher dimensions.

Continuity of Multivariable Functions:

A function f: ℝⁿ → ℝ is continuous at a point a if:

$$\lim_{\mathbf{x} \to \mathbf{a}} f(\mathbf{x}) = f(\mathbf{a})$$

Crucially, convergence x → a must hold for every possible approach path. In single-variable calculus, there are only two directions (left and right). In ℝⁿ, there are infinitely many directions and paths.

Path Dependence of Limits:

Consider the function:

$$f(x, y) = \begin{cases} \frac{xy}{x^2 + y^2} & \text{if } (x, y) eq (0, 0) \ 0 & \text{if } (x, y) = (0, 0) \end{cases}$$

Approaching (0, 0) along the x-axis (y = 0): limit = 0 Approaching along y = x: f(x, x) = x²/(2x²) = 1/2

Different paths give different limits, so this function is not continuous at (0, 0).

Smoothness and Differentiability:

A function is smooth (C∞) if all partial derivatives of all orders exist and are continuous. Most loss functions in machine learning are smooth except at specific points:

• ReLU activation: Non-differentiable at z = 0
• L1 loss / absolute value: Non-differentiable at residual = 0
• Hinge loss: Non-differentiable at margin = 1

These non-smooth points create challenges for gradient-based optimization, motivating smooth approximations (softplus, smooth L1, etc.).

Understanding the structure of the input space is essential for reasoning about optimization and generalization in machine learning.

Open and Closed Sets:

In ℝⁿ, an open ball of radius r centered at c is:

$$B_r(\mathbf{c}) = {\mathbf{x} \in \mathbb{R}^n : |\mathbf{x} - \mathbf{c}| < r}$$

A set S ⊆ ℝⁿ is open if every point has an open ball entirely contained in S. A set is closed if its complement is open (equivalently, if it contains all its limit points).

Bounded and Compact Sets:

A set is bounded if it fits inside some ball of finite radius. A set is compact if it is both closed and bounded. The Extreme Value Theorem states that a continuous function on a compact set attains its maximum and minimum—this guarantees that minimization problems over compact domains have solutions.

Relevance to ML:

Convex Sets:

A set C ⊆ ℝⁿ is convex if the line segment between any two points in C lies entirely within C:

$$\forall \mathbf{x}, \mathbf{y} \in C, \forall t \in [0, 1]: t\mathbf{x} + (1-t)\mathbf{y} \in C$$

Convexity of the domain is crucial for convex optimization, where local minima are global. Examples of convex sets:

• The entire space ℝⁿ
• Any hyperplane or half-space
• Balls (both open and closed)
• The intersection of convex sets
• The probability simplex
• The cone of positive semidefinite matrices

Non-convex domains (e.g., the union of two disjoint balls) can create disconnected feasible regions where gradient descent cannot move between components.

A powerful perspective is to view multivariable functions as transformations that map one space to another. This viewpoint is central to understanding neural networks.

Layer-by-Layer Transformations:

In a neural network, each layer is a function:

$$\mathbf{h}^{(\ell)} = \sigma(\mathbf{W}^{(\ell)} \mathbf{h}^{(\ell-1)} + \mathbf{b}^{(\ell)})$$

This maps from ℝⁿ⁽ˡ⁻¹⁾ (previous layer's dimension) to ℝⁿ⁽ˡ⁾ (current layer's dimension). The full network is the composition of these layer functions:

$$f = f^{(L)} \circ f^{(L-1)} \circ \cdots \circ f^{(1)}$$

The Chain Rule for Compositions:

When we compose functions, their derivatives multiply. If g: ℝⁿ → ℝᵐ and f: ℝᵐ → ℝᵖ, then:

$$\frac{\partial (f \circ g)}{\partial \mathbf{x}} = \frac{\partial f}{\partial \mathbf{g}} \cdot \frac{\partial \mathbf{g}}{\partial \mathbf{x}}$$

This is the matrix chain rule, and it's the mathematical foundation of backpropagation. We'll explore this in depth when we study Jacobian matrices.

Representations and Embeddings:

The intermediate values h⁽ˡ⁾ are learned representations of the input. Each layer transforms the representation, ideally making the final classification or regression task easier. Functions can:

• Expand dimensions: Map ℝⁿ → ℝᵐ where m > n (kernel methods, first hidden layers)
• Contract dimensions: Map ℝⁿ → ℝᵐ where m < n (pooling, bottleneck architectures)
• Preserve dimensions: Map ℝⁿ → ℝⁿ (residual connections, same-size hidden layers)

We've laid the groundwork for multivariate calculus by establishing what multivariable functions are, how to think about them geometrically, and why they're central to machine learning.

Key Concepts Covered:

What's Next:

Now that we understand what multivariable functions are, we're ready to ask: How do they change? The next page introduces gradient vectors—the fundamental tool for measuring how a function changes in all directions simultaneously. The gradient generalizes the derivative to multiple dimensions and provides the direction of steepest ascent, making it the cornerstone of optimization algorithms like gradient descent.