Train-Test Split - Holdout Validation | OneNoughtOne

0/328

Machine LearningCross-Validation & Resampling

Holdout Validation

LevelIntermediate

Duration90 mins

TopicCross-Validation & Resampling

1 / 5

Train-Test Split

The Fundamental Question of Generalization

Every machine learning model faces a single, defining question: How will it perform on data it has never seen before?

This isn't merely an academic concern—it's the very essence of why we build ML systems. A model that memorizes training data perfectly but fails on new data is worthless. A model that generalizes well to unseen examples is valuable. The entire discipline of machine learning revolves around this distinction between memorization and generalization.

The train-test split is our first and most fundamental tool for answering this question. It provides a simple yet powerful methodology: hide some data from the model during training, then reveal how well the model performs on this hidden data. This simple idea—so intuitive that it might seem obvious—forms the bedrock upon which all model evaluation rests.

What You Will Learn

By the end of this page, you will understand the theoretical foundations of train-test splitting, its mathematical justification, optimal split ratios for different scenarios, implementation best practices, and the subtle pitfalls that can invalidate your generalization estimates. You'll gain the precision and depth expected of a Principal ML Engineer.

The Generalization Problem

To understand why train-test splitting is essential, we must first deeply understand the generalization problem in machine learning. This problem arises from a fundamental tension:

We want our model to perform well on future, unseen data (the true objective)
We can only train and evaluate on historical, observed data (what we have access to)

This gap between what we want and what we can measure creates the core challenge. Let's formalize this precisely.

The Statistical Learning Framework

In the statistical learning framework, we assume:

There exists an unknown joint probability distribution $P(X, Y)$ over features $X$ and targets $Y$
Our dataset $\mathcal{D} = {(x_1, y_1), \ldots, (x_n, y_n)}$ consists of i.i.d. samples from this distribution
We seek a function $f: X \rightarrow Y$ that minimizes the expected risk (generalization error):

$$R(f) = \mathbb{E}_{(X,Y) \sim P}[L(Y, f(X))]$$

where $L$ is our loss function (e.g., squared error, cross-entropy).

The critical insight is that we cannot compute $R(f)$ directly—we don't know $P(X,Y)$. We can only compute the empirical risk on our finite sample:

$$\hat{R}n(f) = \frac{1}{n} \sum{i=1}^{n} L(y_i, f(x_i))$$

The Optimism of Training Error

When we evaluate a model on the same data used for training, the empirical risk is systematically optimistic—it underestimates the true generalization error. This is because the model has adapted to the specific noise and idiosyncrasies in the training data. The more flexible the model, the more severe this optimism.

Quantifying the Optimism

The gap between training error and generalization error is precisely what we need to estimate. Let $\hat{f}$ be the model fitted on training data $\mathcal{D}_{train}$. The optimism is defined as:

$$\text{Optimism} = R(\hat{f}) - \hat{R}_{train}(\hat{f})$$

For many models, statistical theory provides bounds on this optimism. For instance, in linear regression with $p$ parameters, the expected optimism is approximately:

$$\mathbb{E}[\text{Optimism}] \approx \frac{2p\sigma^2}{n}$$

where $\sigma^2$ is the noise variance and $n$ is the sample size. This elegant result (related to Mallows' Cp and AIC) quantifies how training error systematically underestimates test error.

However, these theoretical corrections only work for specific model families. For general models—especially complex ones like neural networks or gradient boosting—we need an empirical approach: the train-test split.

The Train-Test Split Methodology

The train-test split methodology is elegant in its simplicity: partition your data into two disjoint subsets, use one for training and one for evaluation. Yet this simplicity belies deep statistical considerations about how to perform this partition optimally.

Formal Definition

Given a dataset $\mathcal{D} = {(x_1, y_1), \ldots, (x_n, y_n)}$, a train-test split partitions the index set ${1, \ldots, n}$ into:

Training indices $\mathcal{I}{train}$ with $|\mathcal{I}{train}| = n_{train}$
Test indices $\mathcal{I}{test}$ with $|\mathcal{I}{test}| = n_{test}$

such that $\mathcal{I}{train} \cap \mathcal{I}{test} = \emptyset$ and $\mathcal{I}{train} \cup \mathcal{I}{test} = {1, \ldots, n}$.

The model $\hat{f}$ is trained on $\mathcal{D}{train} = {(x_i, y_i) : i \in \mathcal{I}{train}}$, and evaluated on $\mathcal{D}{test} = {(x_i, y_i) : i \in \mathcal{I}{test}}$.

The test error then serves as our estimate of generalization error:

$$\hat{R}{test} = \frac{1}{n{test}} \sum_{i \in \mathcal{I}_{test}} L(y_i, \hat{f}(x_i))$$

The Principle of Independence

The validity of test error as a generalization estimate rests on a crucial assumption: the test set must be statistically independent from the training process. Any leakage of test set information into training—even subtle leakage—invalidates the estimate and leads to over-optimistic performance assessments.

Statistical Properties of Test Error

The test error $\hat{R}_{test}$ is a random variable (random because the test set is a random sample). Let's analyze its properties:

Unbiasedness: Under the assumption that test data comes from the same distribution as future data: $$\mathbb{E}[\hat{R}_{test}] = R(\hat{f})$$

The test error is an unbiased estimator of the generalization error for the model trained on that particular training set.

Variance: The variance of the test error estimate is: $$\text{Var}(\hat{R}{test}) = \frac{\text{Var}(L(Y, \hat{f}(X)))}{n{test}}$$

This tells us something critical: larger test sets give more precise estimates. The standard error of our estimate decreases with $1/\sqrt{n_{test}}$.

Confidence Intervals: For sufficiently large test sets, we can construct approximate confidence intervals: $$\hat{R}{test} \pm z{\alpha/2} \cdot \sqrt{\frac{\hat{\sigma}^2_{test}}{n_{test}}}$$

where $\hat{\sigma}^2_{test}$ is the sample variance of losses on the test set.

Split Ratios and Tradeoffs

One of the most common questions practitioners face is: What proportion of data should go into training versus testing? The answer involves navigating a fundamental tradeoff that has profound implications for both model quality and evaluation reliability.

The Bias-Variance Tradeoff in Splitting

The split ratio creates a tension between two competing objectives:

More training data → Better model (lower bias in performance)

The model sees more examples of the underlying pattern
Learning algorithms can extract more nuanced relationships
Especially important for high-capacity models with many parameters

More test data → Better estimate (lower variance in evaluation)

Larger samples give more precise error estimates
Reduces uncertainty about true generalization performance
Enables more reliable model comparisons

Let's quantify this tradeoff mathematically.

Mathematical Analysis of the Tradeoff

Suppose our dataset has $n$ total samples. Let $\alpha$ be the fraction used for testing, so:

$n_{test} = \alpha n$
$n_{train} = (1-\alpha) n$

The estimation variance of our test error is: $$\text{Var}(\hat{R}_{test}) \propto \frac{1}{\alpha n}$$

Meanwhile, the model suboptimality (gap between our model and what we could learn with all data) typically scales as: $$\text{Suboptimality} \propto \frac{1}{(1-\alpha) n}$$

(with the exact exponent depending on the learning algorithm and problem complexity).

The total error in our evaluation can be decomposed as: $$\text{Total Error} \approx \underbrace{\text{Bias from smaller training set}}{f((1-\alpha)n)} + \underbrace{\text{Variance from smaller test set}}{g(\alpha n)}$$

Common Train-Test Split Ratios by Context
Split Ratio	Training %	Test %	Best Use Case	Considerations
90/10	90%	10%	Large datasets (>100k samples)	Maximizes training data; sufficient test samples for precise estimates
80/20	80%	20%	Medium datasets (10k-100k samples)	Classical default; balances both objectives well
70/30	70%	30%	Smaller datasets (1k-10k samples)	Prioritizes evaluation reliability when data is limited
50/50	50%	50%	Very small datasets or model comparison studies	Often suboptimal; prefer cross-validation instead

Rule of Thumb for Test Set Size

A useful heuristic: your test set should have at least 30 samples per class (for classification) or enough samples to give standard errors of ±2-3% on your primary metric. With fewer samples, confidence intervals become so wide that model comparisons lose statistical power.

Modern Perspective: The 'Enough Data' Threshold

In the era of big data, the tradeoff looks different. With millions of samples:

Even 1% held out for testing might be 10,000+ samples—plenty for precise estimates
The marginal benefit of additional training samples diminishes (learning curves flatten)
The primary constraint becomes computational, not statistical

For deep learning on massive datasets, splits like 98/1/1 (train/validation/test) are common. The key insight: once you have 'enough' test samples for your precision needs, additional test data doesn't help much, so put it in training.

Conversely, with small datasets (<1,000 samples), even generous test sets have high variance. This is where cross-validation becomes essential—but that's a topic for the next module.

Implementation Best Practices

Implementing train-test splits correctly requires attention to several technical details. Mistakes here can lead to data leakage, biased estimates, or irreproducible results. Let's examine the gold-standard practices used in production ML systems.

train_test_split_examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
import numpy as np
from sklearn.model_selection import train_test_split
 
# ============================================
# Basic Train-Test Split
# ============================================
X = np.random.randn(1000, 10)  # 1000 samples, 10 features
y = np.random.randint(0, 2, 1000)  # Binary classification
 
# Standard 80/20 split with reproducibility
X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,      # 20% for testing
    random_state=42,    # CRITICAL: Set seed for reproducibility
    shuffle=True        # Default, but explicit is better
)
 
print(f"Training set: {X_train.shape[0]} samples")
print(f"Test set: {X_test.shape[0]} samples")
 
# ============================================
# Stratified Split (Preserves Class Distribution)
# ============================================
# When classes are imbalanced, stratification is essential
y_imbalanced = np.array([0]*900 + [1]*100)  # 90/10 class imbalance
 
X_train_strat, X_test_strat, y_train_strat, y_test_strat = train_test_split(
    X, y_imbalanced,
    test_size=0.2,
    random_state=42,
    stratify=y_imbalanced  # CRITICAL for imbalanced data
)
 
# Verify stratification maintained class proportions
train_ratio = y_train_strat.mean()
test_ratio = y_test_strat.mean()
print(f"Training minority ratio: {train_ratio:.3f}")
print(f"Test minority ratio: {test_ratio:.3f}")
# Should both be ~0.10
 
# ============================================
# Multi-Output Stratification
# ============================================
# For multi-label or multi-output problems
from sklearn.model_selection import StratifiedShuffleSplit
 
# Custom stratification for complex scenarios
sss = StratifiedShuffleSplit(
    n_splits=1,
    test_size=0.2,
    random_state=42
)
 
for train_idx, test_idx in sss.split(X, y):
    X_train_custom = X[train_idx]
    X_test_custom = X[test_idx]
    y_train_custom = y[train_idx]
    y_test_custom = y[test_idx]
 
# ============================================
# Index-Based Splitting (Recommended for DataFrames)
# ============================================
import pandas as pd
 
df = pd.DataFrame({
    'feature_1': np.random.randn(1000),
    'feature_2': np.random.randn(1000),
    'target': np.random.randint(0, 2, 1000)
})
 
# Create explicit index arrays for maximum control
indices = np.arange(len(df))
train_indices, test_indices = train_test_split(
    indices,
    test_size=0.2,
    random_state=42
)
 
# Split using indices (preserves DataFrame structure)
df_train = df.iloc[train_indices].copy()
df_test = df.iloc[test_indices].copy()
 
# Store indices for audit trail
df_train['_split'] = 'train'
df_test['_split'] = 'test'

Critical Implementation Guidelines

•Always set random seeds — Reproducibility is non-negotiable in production ML. Document the seed in code, configuration, and experiment tracking.
•Use stratification for classification — Especially critical with imbalanced classes or small datasets. Ensures both splits reflect the true class distribution.
•Split before any preprocessing — Never fit scalers, encoders, or any transformation on the full dataset. Fit only on training data, then transform test data.
•Preserve indices for auditing — Store which samples went into which split. Essential for debugging and regulatory compliance.
•Never look at test data during development — The test set should be truly held out until final evaluation. Multiple peeks invalidate the estimate.

Random Shuffling and Its Implications

The seemingly simple decision to 'shuffle the data before splitting' carries profound implications. Understanding when to shuffle—and critically, when not to—separates competent practitioners from those who unknowingly corrupt their evaluation.

Why We Shuffle: Breaking Spurious Correlations

Datasets often arrive ordered in ways that create problems:

Collection order bias: Data collected early may systematically differ from data collected later (drift, different conditions)
Sorting artifacts: Data exported from databases may be sorted by ID, time, or other fields
Grouping effects: Similar samples may be clustered together (all fraud cases first, then legitimate cases)

Without shuffling, a simple temporal split could give a test set that's systematically different from the training set—not because of true distribution shift, but because of collection artifacts.

The Shuffle Process Mathematically

Shuffling creates a random permutation $\pi$ of indices ${1, \ldots, n}$. The first $n_{train}$ elements of $\pi$ become training indices; the rest become test indices. Under uniform random shuffling:

$$P(i \in \mathcal{I}{test}) = \frac{n{test}}{n} = \alpha \quad \forall i$$

Every sample has equal probability of being in the test set, which is the foundation of unbiased sampling.

When NOT to Shuffle: Temporal Data

For time series data or any data with temporal dependencies, random shuffling is DANGEROUS. It creates 'future leakage'—using future information to predict the past. Always use temporal splits for sequential data: train on the past, test on the future.

When to Shuffle

•Cross-sectional data (independent samples at one time point)
•Image classification with independent images
•Tabular data without temporal or group structure
•Text classification with independent documents
•When sample order is arbitrary/administrative

When NOT to Shuffle

•Time series forecasting (any temporal prediction)
•Financial data with temporal dependencies
•Sequential user behavior prediction
•When detecting distribution drift is important
•When future data shouldn't inform past predictions

Grouped Data: A Special Consideration

When data contains groups (e.g., multiple samples from same patient, multiple frames from same video, multiple transactions from same user), shuffling at the sample level creates another form of leakage:

Training and test might contain different samples from the same group
The model learns group-specific patterns that won't generalize
Test performance is artificially inflated

Solution: Use group-aware splitting. Entire groups must go into either training or testing, never split across both. This is covered in detail in the module on Group Cross-Validation.

The Data Leakage Problem

Data leakage is the most insidious failure mode in machine learning evaluation. It occurs when information from the test set inappropriately influences the training process or model selection. The result: evaluation metrics that look excellent but don't reflect real-world performance.

Types of Data Leakage in Train-Test Splitting

1. Preprocessing Leakage The most common form: fitting transformations on the full dataset before splitting.

# WRONG - leakage through scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # Fitted on ALL data including test
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, ...)

# RIGHT - fit only on training data
X_train, X_test, y_train, y_test = train_test_split(X, y, ...)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Fit only on train
X_test_scaled = scaler.transform(X_test)        # Transform test

The leaked information: test set statistics (mean, variance) influence training. While this seems minor for scaling, it becomes severe for feature selection or imputation.

2. Feature Engineering Leakage Creating features using information from the test set:

Target encoding computed on full data
PCA components computed on full feature matrix
Feature selection based on full-data correlations

3. Temporal Leakage Using future information in features:

Features computed from full time series (moving averages spanning train/test boundary)
External data joined without temporal awareness

4. Selection Leakage Choosing the model or hyperparameters based on test performance, then 'evaluating' on the same test set.

The Magnitude of Leakage Impact

Leakage doesn't just add noise—it systematically inflates performance. In Kaggle competitions, top solutions with data leakage have dropped from 1st place to 1000th when leakage was removed. In production, models with leakage often fail catastrophically on truly new data.

preventing_leakage
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
 
# ============================================
# Leakage-Free Pipeline Pattern
# ============================================
# The key insight: ALL preprocessing must be inside the pipeline
# The pipeline is fitted only on training data
 
# Split FIRST, before any processing
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
 
# Build pipeline with all preprocessing steps
pipeline = Pipeline([
    ('scaler', StandardScaler()),           # Fitted on train only
    ('feature_selection', SelectKBest(      # Fitted on train only
        score_func=f_classif,
        k=5
    )),
    ('classifier', LogisticRegression())    # Fitted on train only
])
 
# Fit entire pipeline on training data only
pipeline.fit(X_train, y_train)
 
# Evaluate on test data - preprocessing uses train-fitted transforms
test_score = pipeline.score(X_test, y_test)
 
# ============================================
# Detecting Leakage: Sanity Checks
# ============================================
def leakage_sanity_check(train_error, test_error, baseline):
    """
    Warning signs that suggest potential leakage:
    1. Test error much lower than expected for problem complexity
    2. Test error surprisingly close to train error (low gap)
    3. Performance doesn't degrade on truly new data
    """
    gap = train_error - test_error
    
    warnings = []
    
    if test_error < baseline * 0.5:
        warnings.append("Test error suspiciously low vs baseline")
    
    if abs(gap) < 0.01:  # Near-perfect generalization
        warnings.append("Train-test gap suspiciously small")
    
    if gap < 0:  # Test better than train
        warnings.append("CRITICAL: Test error lower than train error")
    
    return warnings
 
# ============================================
# Temporal Leakage Prevention
# ============================================
import pandas as pd
 
def temporal_train_test_split(df, date_col, test_start_date):
    """
    Split data temporally: train on past, test on future.
    No shuffling - respects temporal order.
    """
    df = df.sort_values(date_col)
    
    train_mask = df[date_col] < test_start_date
    test_mask = df[date_col] >= test_start_date
    
    # Verify no overlap
    assert (train_mask & test_mask).sum() == 0, "Overlap detected!"
    
    return df[train_mask].copy(), df[test_mask].copy()

Evaluating Split Quality

A good train-test split should produce training and test sets that are statistically similar (representing the same underlying distribution). Several diagnostic techniques help verify split quality.

Distribution Comparison Tests

For each feature, compare distributions between train and test:

Continuous Features:

Kolmogorov-Smirnov test: Tests if two samples come from same distribution
Wasserstein distance: Measures 'earth mover's distance' between distributions
Visual: Overlaid density plots or Q-Q plots

Categorical Features:

Chi-square test: Compares frequency distributions
Proportion comparisons with confidence intervals
Visual: Side-by-side bar charts

Target Variable:

Classification: Compare class proportions (should match if stratified)
Regression: Compare mean, median, variance, and distribution shape

split_quality_diagnostics
Python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
import numpy as np
import pandas as pd
from scipy import stats
from scipy.stats import ks_2samp, chi2_contingency
import matplotlib.pyplot as plt
 
def diagnose_split_quality(train_df, test_df, target_col=None):
    """
    Comprehensive diagnostics for train-test split quality.
    Returns warnings if significant distribution differences detected.
    """
    results = {'warnings': [], 'feature_analysis': {}}
    
    # Get common columns
    common_cols = set(train_df.columns) & set(test_df.columns)
    if target_col:
        common_cols.discard(target_col)
    
    for col in common_cols:
        train_data = train_df[col].dropna()
        test_data = test_df[col].dropna()
        
        if train_df[col].dtype in ['int64', 'float64']:
            # Continuous feature: KS test
            ks_stat, ks_pval = ks_2samp(train_data, test_data)
            
            results['feature_analysis'][col] = {
                'type': 'continuous',
                'ks_statistic': ks_stat,
                'ks_pvalue': ks_pval,
                'train_mean': train_data.mean(),
                'test_mean': test_data.mean(),
                'train_std': train_data.std(),
                'test_std': test_data.std()
            }
            
            if ks_pval < 0.01:  # Significant difference
                results['warnings'].append(
                    f"Feature '{col}': Significant distribution difference "
                    f"(KS p-value: {ks_pval:.4f})"
                )
        
        else:
            # Categorical feature: Chi-square test
            train_counts = train_data.value_counts()
            test_counts = test_data.value_counts()
            
            # Align categories
            all_cats = set(train_counts.index) | set(test_counts.index)
            train_aligned = [train_counts.get(c, 0) for c in all_cats]
            test_aligned = [test_counts.get(c, 0) for c in all_cats]
            
            if len(all_cats) > 1 and min(train_aligned) > 0 and min(test_aligned) > 0:
                contingency = np.array([train_aligned, test_aligned])
                chi2, chi2_pval, _, _ = chi2_contingency(contingency)
                
                results['feature_analysis'][col] = {
                    'type': 'categorical',
                    'chi2_statistic': chi2,
                    'chi2_pvalue': chi2_pval,
                    'n_categories': len(all_cats)
                }
                
                if chi2_pval < 0.01:
                    results['warnings'].append(
                        f"Feature '{col}': Significant distribution difference "
                        f"(Chi2 p-value: {chi2_pval:.4f})"
                    )
    
    # Target analysis
    if target_col:
        train_target = train_df[target_col]
        test_target = test_df[target_col]
        
        if train_target.dtype in ['int64', 'float64'] and train_target.nunique() > 10:
            # Regression target
            results['target_analysis'] = {
                'train_mean': train_target.mean(),
                'test_mean': test_target.mean(),
                'train_std': train_target.std(),
                'test_std': test_target.std()
            }
        else:
            # Classification target
            train_dist = train_target.value_counts(normalize=True).to_dict()
            test_dist = test_target.value_counts(normalize=True).to_dict()
            results['target_analysis'] = {
                'train_distribution': train_dist,
                'test_distribution': test_dist
            }
            
            # Check for significant class proportion differences
            for cls in set(train_dist.keys()) | set(test_dist.keys()):
                train_prop = train_dist.get(cls, 0)
                test_prop = test_dist.get(cls, 0)
                if abs(train_prop - test_prop) > 0.05:
                    results['warnings'].append(
                        f"Target class '{cls}': Proportion differs by "
                        f"{abs(train_prop - test_prop):.2%}"
                    )
    
    return results
 
# Example usage
results = diagnose_split_quality(df_train, df_test, target_col='target')
print(f"Warnings found: {len(results['warnings'])}")
for warning in results['warnings']:
    print(f"  ⚠️ {warning}")

The Adversarial Validation Technique

A powerful technique: train a classifier to distinguish train from test samples. If the classifier achieves AUC >> 0.5, the splits are too different. Features with high importance in this classifier indicate distribution mismatch. This technique also detects covariate shift between training and production data.

Summary and Key Takeaways

The train-test split is foundational to honest model evaluation. Let's consolidate the essential principles:

Core Principles

•Test error estimates generalization — The test set serves as a proxy for future unseen data, providing an unbiased estimate of real-world performance.
•Split ratios involve tradeoffs — More training data improves the model; more test data improves the estimate. Balance based on dataset size and precision needs.
•Random shuffling breaks collection artifacts — Shuffle i.i.d. data to ensure representative splits, but never shuffle temporal or grouped data.
•Leakage invalidates everything — Any information flow from test to training corrupts your estimate. Use pipelines and strict separation.
•Reproducibility is mandatory — Always set random seeds and document split procedures. Irreproducible experiments are unscientific.
•Verify split quality — Use statistical tests to confirm train and test distributions match. Catch problems before they corrupt downstream analysis.

The Bigger Picture

While the train-test split is fundamental, it has significant limitations:

High variance: A single split may not represent the full data variability
Wasteful: Test data can never be used for training
No hyperparameter tuning: Using test set for tuning invalidates the estimate

These limitations motivate the validation set (covered next) and cross-validation (covered in Module 2). The train-test split remains essential as the final, unbiased evaluation step—but it's just one piece of a complete model development workflow.

Page Complete

You now understand the train-test split at a depth suitable for production ML systems. Next, we'll explore the validation set—a critical addition that enables hyperparameter tuning without compromising test set integrity.