OneNoughtOne
0/328
Loading content...
Read aloud is unavailable in this browser.
Every machine learning model operates at two distinct levels of abstraction. At the first level, the model learns parameters from data—the weights in a neural network, the coefficients in a linear regression, or the split points in a decision tree. At the second level, we choose hyperparameters that govern how that learning happens—the learning rate, the number of hidden layers, or the tree depth.
This distinction between parameters and hyperparameters is fundamental to understanding modern machine learning. Yet it's often glossed over in introductory treatments, leading to confusion about what we're actually optimizing and how. In this page, we'll develop a rigorous understanding of this distinction—one that will serve as the foundation for all hyperparameter optimization techniques.
By the end of this page, you will: • Precisely distinguish model parameters from hyperparameters • Understand why this distinction creates a bi-level optimization problem • Recognize hyperparameters across different model families • Appreciate the fundamental challenge that hyperparameter optimization addresses
Let's establish precise definitions that will guide our thinking throughout this chapter:
Model Parameters are the internal variables that a model learns from the training data. They define the mapping from inputs to outputs that the model has discovered. Parameters exist within the model's hypothesis space and are optimized directly by the learning algorithm.
Hyperparameters are configuration settings that define how the learning process works. They exist outside the model and cannot be learned from the training data directly. Hyperparameters determine the hypothesis space itself, the learning dynamics, or the regularization strength.
The key insight is that parameters are learned; hyperparameters are chosen. This creates a hierarchical optimization problem: we must choose hyperparameters before learning can begin, but we can only evaluate hyperparameter choices after learning completes.
| Characteristic | Model Parameters | Hyperparameters |
|---|---|---|
| Definition | Internal model variables | External configuration settings |
| Determined by | Learning algorithm (gradient descent, etc.) | Practitioner or optimization algorithm |
| Learned from | Training data | Cannot be learned from data directly |
| When set | During training | Before training begins |
| Evaluation requires | Forward pass through model | Complete training run + validation |
| Scale | Can be millions to billions | Typically dozens to hundreds |
| Gradient signal | Direct gradient from loss | No direct gradient (typically) |
From a meta-learning perspective, hyperparameters encode our prior beliefs about what learning processes work well. When we choose a learning rate of 0.001, we're expressing a belief that updates should be small. When we set max_depth=10, we're limiting model complexity based on prior experience. Hyperparameter optimization can be viewed as learning these priors from data.
To reason precisely about hyperparameter optimization, we need a formal framework. Let's develop the mathematical notation that underlies all HPO methods.
Consider a learning algorithm $\mathcal{A}$ that takes training data $\mathcal{D}_{\text{train}}$ and produces a model with parameters $\theta$. The algorithm is parameterized by hyperparameters $\lambda \in \Lambda$, where $\Lambda$ is the hyperparameter space:
$$\theta^*(\lambda) = \mathcal{A}(\mathcal{D}_{\text{train}}, \lambda)$$
The learned parameters $\theta^*$ depend on our hyperparameter choice $\lambda$. Different hyperparameters lead to different learned models.
The inner optimization (training) finds the best parameters for fixed hyperparameters:
$$\theta^*(\lambda) = \arg\min_\theta \mathcal{L}_{\text{train}}(\theta, \lambda)$$
The outer optimization (hyperparameter optimization) finds the best hyperparameters based on validation performance:
$$\lambda^* = \arg\min_\lambda \mathcal{L}_{\text{val}}(\theta^*(\lambda))$$
This formulation reveals why HPO is fundamentally difficult: the outer objective depends on the solution to the inner problem. Evaluating a single hyperparameter configuration requires solving an entire training problem. With modern deep learning models, a single evaluation can take hours to days.
The Response Surface
The function $f(\lambda) = \mathcal{L}_{\text{val}}(\theta^*(\lambda))$ that maps hyperparameters to validation performance is called the response surface or objective landscape. This surface has several challenging properties:
Non-differentiable: We typically cannot compute $\frac{\partial \mathcal{L}_{\text{val}}}{\partial \lambda}$ directly because training is a complex, discrete procedure
Expensive to evaluate: Each point on the surface requires a full training run
Noisy: Stochastic elements in training (mini-batch sampling, initialization) make evaluations noisy
Non-convex: The surface typically has multiple local minima and complex structure
High-dimensional: Many hyperparameters create a high-dimensional search space
These properties explain why naive hyperparameter optimization (random guessing, manual tuning) often performs poorly, and why specialized optimization techniques have been developed.
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185
"""Bi-level Optimization Framework for Hyperparameter Optimization This illustrates the mathematical structure of HPO as a bi-level problem."""import numpy as npfrom typing import Callable, Dict, Any, Tuple class BiLevelOptimizer: """ Demonstrates the bi-level structure of hyperparameter optimization. Outer level: Optimize hyperparameters λ Inner level: Optimize model parameters θ given λ """ def __init__( self, inner_optimizer: Callable[[Dict, np.ndarray], np.ndarray], train_loss: Callable[[np.ndarray, Dict], float], val_loss: Callable[[np.ndarray], float], lambda_space: Dict[str, Tuple[float, float]], ): """ Args: inner_optimizer: A(D_train, λ) → θ* - training algorithm train_loss: L_train(θ, λ) - training objective val_loss: L_val(θ) - validation objective lambda_space: Hyperparameter bounds {name: (low, high)} """ self.inner_optimizer = inner_optimizer self.train_loss = train_loss self.val_loss = val_loss self.lambda_space = lambda_space # Track all evaluations (expensive, so we cache) self.evaluation_history = [] def evaluate_hyperparameters( self, lambda_config: Dict[str, float], train_data: np.ndarray, ) -> float: """ Evaluate a single hyperparameter configuration. This is the EXPENSIVE step in HPO: 1. Run complete inner optimization (training) 2. Evaluate learned model on validation set Args: lambda_config: Hyperparameter values train_data: Training dataset Returns: Validation loss for this hyperparameter configuration """ # Inner optimization: θ*(λ) = A(D_train, λ) # This is where the computational cost lives theta_star = self.inner_optimizer(lambda_config, train_data) # Outer objective evaluation: L_val(θ*(λ)) val_performance = self.val_loss(theta_star) # Cache the result self.evaluation_history.append({ 'lambda': lambda_config.copy(), 'theta': theta_star.copy(), 'val_loss': val_performance, }) return val_performance def response_surface_at(self, lambda_config: Dict) -> float: """ Evaluate the response surface f(λ) = L_val(θ*(λ)). The response surface is what we're trying to minimize, but each evaluation requires a full training run. """ # Check cache first (important optimization) for entry in self.evaluation_history: if entry['lambda'] == lambda_config: return entry['val_loss'] # Cache miss: must retrain raise ValueError("Configuration not yet evaluated - requires training") def estimate_gradient( self, lambda_config: Dict, epsilon: float = 1e-3, ) -> Dict[str, float]: """ Estimate gradient of response surface via finite differences. This shows why gradient-based HPO is expensive: Each dimension requires 2 full training runs! For d hyperparameters: 2d training runs per gradient estimate. """ gradients = {} for name in lambda_config: # Evaluate f(λ + ε*e_i) lambda_plus = lambda_config.copy() lambda_plus[name] += epsilon f_plus = self.evaluate_hyperparameters(lambda_plus, None) # Evaluate f(λ - ε*e_i) lambda_minus = lambda_config.copy() lambda_minus[name] -= epsilon f_minus = self.evaluate_hyperparameters(lambda_minus, None) # Central difference approximation gradients[name] = (f_plus - f_minus) / (2 * epsilon) return gradients # Example: Linear Regression with L2 Regularizationdef linear_regression_example(): """ Demonstrates parameters vs hyperparameters in ridge regression. Parameters (θ): Regression coefficients w, bias b Hyperparameters (λ): Regularization strength α Inner problem: min_θ ||Xθ - y||² + α||θ||² Outer problem: min_α L_val(θ*(α)) """ np.random.seed(42) # Generate synthetic data n_samples, n_features = 100, 10 X_train = np.random.randn(n_samples, n_features) X_val = np.random.randn(50, n_features) true_weights = np.random.randn(n_features) y_train = X_train @ true_weights + 0.1 * np.random.randn(n_samples) y_val = X_val @ true_weights + 0.1 * np.random.randn(50) def inner_optimizer(config: Dict, X: np.ndarray) -> np.ndarray: """ Solve ridge regression: θ* = (X'X + αI)^(-1) X'y This is the inner optimization - finding optimal θ for given α. """ alpha = config['regularization_strength'] # Closed-form solution for ridge regression XtX = X_train.T @ X_train Xty = X_train.T @ y_train theta = np.linalg.solve(XtX + alpha * np.eye(n_features), Xty) return theta def val_loss(theta: np.ndarray) -> float: """Validation MSE - what we ultimately want to minimize.""" predictions = X_val @ theta return np.mean((predictions - y_val) ** 2) # Explore the response surface print("Exploring response surface: Validation Loss vs Regularization") print("=" * 60) alphas = [1e-4, 1e-3, 1e-2, 1e-1, 1.0, 10.0, 100.0] for alpha in alphas: config = {'regularization_strength': alpha} theta_star = inner_optimizer(config, X_train) val_mse = val_loss(theta_star) print(f"α = {alpha:>8.4f} → θ* norm = {np.linalg.norm(theta_star):>8.4f}" f" → Val MSE = {val_mse:.6f}") # Find optimal α best_alpha = min(alphas, key=lambda a: val_loss( inner_optimizer({'regularization_strength': a}, X_train) )) print(f"Optimal regularization: α* = {best_alpha}") print("Note: Each row above required solving a complete optimization problem!") if __name__ == "__main__": linear_regression_example()Different model families have different hyperparameters, but they serve similar conceptual roles. Let's survey the hyperparameter landscape across common ML models, categorizing them by their function.
Linear and Logistic Regression
Linear models have few hyperparameters, making them an excellent starting point for understanding the concept.
Parameters (Learned):
Hyperparameters (Chosen):
As α → 0, the model approaches unregularized least squares (high variance, low bias). As α → ∞, coefficients shrink toward zero (low variance, high bias). The optimal α balances these extremes for your specific data.
While the parameter/hyperparameter distinction is conceptually clean, real-world ML systems sometimes blur this boundary. Understanding these edge cases deepens our appreciation of the underlying concepts.
When in doubt, ask: "Can this be learned via gradient descent on the training loss?" If yes, it's a parameter. If it requires validation data or is set before training, it's a hyperparameter. If it's learned but requires validation, you're in meta-learning territory.
The Hypergradient Perspective
Recent research has developed hypergradient methods that compute gradients of validation loss with respect to hyperparameters. This is achieved through techniques like:
These methods begin to treat hyperparameters more like parameters, optimizing them via gradient descent. However, they don't eliminate the fundamental distinction—they just make the outer optimization more efficient.
12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576
"""Hypergradient: Gradients of Hyperparameters This illustrates how hypergradients can be computed, showing that evenwhen we can differentiate through training, the bi-level structure remains."""import torchimport torch.nn as nn def unrolled_training(model, train_loader, val_loader, lr, steps=100): """ Perform training while maintaining computation graph for hypergradient. The key insight: if we don't detach intermediate states, we can backpropagate through the entire training process. """ # Learning rate as a differentiable tensor learning_rate = torch.tensor(lr, requires_grad=True) # Create a fresh model copy for this training run model_copy = copy_model(model) optimizer_state = None for step in range(steps): # Forward pass (training) for x_batch, y_batch in train_loader: loss = compute_loss(model_copy, x_batch, y_batch) # Manual SGD step (to maintain graph) grads = torch.autograd.grad(loss, model_copy.parameters(), create_graph=True) # Update parameters (keeping in graph!) for param, grad in zip(model_copy.parameters(), grads): param.data = param - learning_rate * grad # Note: param.data breaks the graph # For true hypergradients, we'd need param = param - lr * grad # Now compute validation loss val_loss = compute_validation_loss(model_copy, val_loader) # This gradient exists because we maintained the computation graph # through all training steps hypergradient = torch.autograd.grad(val_loss, learning_rate) return val_loss, hypergradient def approximate_hypergradient(model, train_data, val_data, lambda_val, epsilon=0.01): """ Approximate hypergradient via finite differences. d(val_loss)/d(lambda) ≈ [val_loss(lambda + ε) - val_loss(lambda - ε)] / 2ε Note: This requires 2 complete training runs per hyperparameter dimension! """ # Training with lambda + epsilon model_plus = train_model(model, train_data, lambda_val + epsilon) val_loss_plus = evaluate(model_plus, val_data) # Training with lambda - epsilon model_minus = train_model(model, train_data, lambda_val - epsilon) val_loss_minus = evaluate(model_minus, val_data) # Finite difference approximation approx_hypergradient = (val_loss_plus - val_loss_minus) / (2 * epsilon) return approx_hypergradient # Key insight: Even with hypergradients, we still have:# 1. An inner optimization (training)# 2. An outer optimization (hyperparameter tuning)# # Hypergradients just make the outer optimization gradient-based# instead of black-box, but don't eliminate the bi-level structure.Understanding the parameter/hyperparameter distinction has direct implications for how you approach machine learning projects:
1. Budget Allocation
With limited compute budget, you must decide how to allocate resources between:
The optimal allocation depends on whether you're in the "undertrained" regime (more training helps) or "hyperparameter-limited" regime (better hyperparameters help more).
2. Evaluation Protocol Design
The distinction dictates your evaluation strategy:
Violating this separation leads to selection bias—your reported performance will be optimistically biased.
3. Reproducibility
For results to be reproducible, you must report:
Without this information, others cannot replicate your results, and your reported performance may not be achievable.
4. Transfer and Generalization
Optimal hyperparameters often don't transfer:
This is why hyperparameter tuning is not a one-time cost—it's recurring as your data and infrastructure evolve.
Published ML results often underreport the computational cost of hyperparameter tuning. A paper might say 'training took 2 hours' but hide the 200 hours spent finding those hyperparameters. When estimating project timelines and costs, always account for HPO.
The history of hyperparameter optimization reflects the broader evolution of machine learning from an art to a science.
The Manual Era (Pre-2000s)
Early ML practitioners relied on intuition, rules of thumb, and extensive experimentation. Hyperparameter selection was treated as craft knowledge, passed down through papers, textbooks, and mentorship. This worked when models were simple and training was fast.
The Grid Search Era (2000s)
As models grew more complex, practitioners formalized hyperparameter search. Grid search became standard: define a grid of values, train a model for each combination, select the best. This was systematic but exponentially expensive.
The Random Search Revolution (2012)
Bergstra and Bengio's 2012 paper showed that random search often outperforms grid search with the same budget. This was counterintuitive but mathematically sound: random search explores more values of important hyperparameters.
The Bayesian Optimization Era (2013+)
BAYESopt, SMAC, and TPE brought principled sequential optimization. By modeling the response surface, these methods could make intelligent decisions about where to evaluate next, dramatically reducing the number of required training runs.
The Modern Era (2018+)
Today's HPO landscape includes:
| Era | Method | Evaluations for 10D Space | Key Insight |
|---|---|---|---|
| Manual | Expert intuition | ~10-50 | Domain knowledge matters |
| Grid | Exhaustive grid | 3¹⁰ = 59,049 | Systematic but expensive |
| Random | Random sampling | ~100-1000 | Focus on important dimensions |
| Bayesian | Sequential optimization | ~50-200 | Model guidance reduces waste |
| Multi-fidelity | Early stopping | ~100-500 (partial) | Cheap evaluations via early stopping |
We've established the fundamental concepts that underpin all of hyperparameter optimization. Let's consolidate the key insights:
What's Next
With this foundation in place, we'll next examine how to define the search space for hyperparameter optimization. This seemingly simple task—specifying what values hyperparameters can take—has profound implications for HPO efficiency and success.
You now understand the fundamental distinction between model parameters and hyperparameters, and why this creates the bi-level optimization problem that HPO addresses. This conceptual foundation will inform everything that follows in this chapter.