7a:T42f,The train-test split methodology is elegant in its simplicity: partition your data into two disjoint subsets, use one for training and one for evaluation. Yet this simplicity belies deep statistical considerations about how to perform this partition optimally. **Formal Definition** Given a dataset $\mathcal{D} = \{(x_1, y_1), \ldots, (x_n, y_n)\}$, a train-test split partitions the index set $\{1, \ldots, n\}$ into: - Training indices $\mathcal{I}_{train}$ with $|\mathcal{I}_{train}| = n_{train}$ - Test indices $\mathcal{I}_{test}$ with $|\mathcal{I}_{test}| = n_{test}$ such that $\mathcal{I}_{train} \cap \mathcal{I}_{test} = \emptyset$ and $\mathcal{I}_{train} \cup \mathcal{I}_{test} = \{1, \ldots, n\}$. The model $\hat{f}$ is trained on $\mathcal{D}_{train} = \{(x_i, y_i) : i \in \mathcal{I}_{train}\}$, and evaluated on $\mathcal{D}_{test} = \{(x_i, y_i) : i \in \mathcal{I}_{test}\}$. The **test error** then serves as our estimate of generalization error: $$\hat{R}_{test} = \frac{1}{n_{test}} \sum_{i \in \mathcal{I}_{test}} L(y_i, \hat{f}(x_i))$$7b:Ta64,import numpy as np from sklearn.model_selection import train_test_split # ============================================ # Basic Train-Test Split # ============================================ X = np.random.randn(1000, 10) # 1000 samples, 10 features y = np.random.randint(0, 2, 1000) # Binary classification # Standard 80/20 split with reproducibility X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, # 20% for testing random_state=42, # CRITICAL: Set seed for reproducibility shuffle=True # Default, but explicit is better ) print(f"Training set: {X_train.shape[0]} samples") print(f"Test set: {X_test.shape[0]} samples") # ============================================ # Stratified Split (Preserves Class Distribution) # ============================================ # When classes are imbalanced, stratification is essential y_imbalanced = np.array([0]*900 + [1]*100) # 90/10 class imbalance X_train_strat, X_test_strat, y_train_strat, y_test_strat = train_test_split( X, y_imbalanced, test_size=0.2, random_state=42, stratify=y_imbalanced # CRITICAL for imbalanced data ) # Verify stratification maintained class proportions train_ratio = y_train_strat.mean() test_ratio = y_test_strat.mean() print(f"Training minority ratio: {train_ratio:.3f}") print(f"Test minority ratio: {test_ratio:.3f}") # Should both be ~0.10 # ============================================ # Multi-Output Stratification # ============================================ # For multi-label or multi-output problems from sklearn.model_selection import StratifiedShuffleSplit # Custom stratification for complex scenarios sss = StratifiedShuffleSplit( n_splits=1, test_size=0.2, random_state=42 ) for train_idx, test_idx in sss.split(X, y): X_train_custom = X[train_idx] X_test_custom = X[test_idx] y_train_custom = y[train_idx] y_test_custom = y[test_idx] # ============================================ # Index-Based Splitting (Recommended for DataFrames) # ============================================ import pandas as pd df = pd.DataFrame({ 'feature_1': np.random.randn(1000), 'feature_2': np.random.randn(1000), 'target': np.random.randint(0, 2, 1000) }) # Create explicit index arrays for maximum control indices = np.arange(len(df)) train_indices, test_indices = train_test_split( indices, test_size=0.2, random_state=42 ) # Split using indices (preserves DataFrame structure) df_train = df.iloc[train_indices].copy() df_test = df.iloc[test_indices].copy() # Store indices for audit trail df_train['_split'] = 'train' df_test['_split'] = 'test'7c:T5c2,library(caret) library(rsample) # Generate sample data set.seed(42) n <- 1000 X <- matrix(rnorm(n * 10), ncol = 10) y <- factor(sample(0:1, n, replace = TRUE)) data <- data.frame(X, target = y) # ============================================ # Basic Train-Test Split with caret # ============================================ # createDataPartition maintains class distribution by default train_index <- createDataPartition( y = data$target, p = 0.8, # 80% for training list = FALSE # Return index vector ) train_data <- data[train_index, ] test_data <- data[-train_index, ] cat(sprintf("Training set: %d samples ", nrow(train_data))) cat(sprintf("Test set: %d samples ", nrow(test_data))) # ============================================ # Using rsample for Tidyverse workflow # ============================================ library(rsample) # Initial split with stratification set.seed(42) data_split <- initial_split( data, prop = 0.8, strata = target # Stratify by target variable ) train_data_rsample <- training(data_split) test_data_rsample <- testing(data_split) # ============================================ # Verify Class Distribution # ============================================ cat(" Class distribution verification: ") cat(sprintf("Original: %.3f ", mean(as.numeric(data$target) - 1))) cat(sprintf("Training: %.3f ", mean(as.numeric(train_data$target) - 1))) cat(sprintf("Test: %.3f ", mean(as.numeric(test_data$target) - 1)))7d:T43c,**Why We Shuffle: Breaking Spurious Correlations** Datasets often arrive ordered in ways that create problems: - **Collection order bias:** Data collected early may systematically differ from data collected later (drift, different conditions) - **Sorting artifacts:** Data exported from databases may be sorted by ID, time, or other fields - **Grouping effects:** Similar samples may be clustered together (all fraud cases first, then legitimate cases) Without shuffling, a simple temporal split could give a test set that's systematically different from the training set—not because of true distribution shift, but because of collection artifacts. **The Shuffle Process Mathematically** Shuffling creates a random permutation $\pi$ of indices $\{1, \ldots, n\}$. The first $n_{train}$ elements of $\pi$ become training indices; the rest become test indices. Under uniform random shuffling: $$P(i \in \mathcal{I}_{test}) = \frac{n_{test}}{n} = \alpha \quad \forall i$$ Every sample has equal probability of being in the test set, which is the foundation of unbiased sampling.7e:T58c,**Types of Data Leakage in Train-Test Splitting** **1. Preprocessing Leakage** The most common form: fitting transformations on the full dataset before splitting. ``` # WRONG - leakage through scaling from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # Fitted on ALL data including test X_train, X_test, y_train, y_test = train_test_split(X_scaled, y, ...) # RIGHT - fit only on training data X_train, X_test, y_train, y_test = train_test_split(X, y, ...) scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) # Fit only on train X_test_scaled = scaler.transform(X_test) # Transform test ``` The leaked information: test set statistics (mean, variance) influence training. While this seems minor for scaling, it becomes severe for feature selection or imputation. **2. Feature Engineering Leakage** Creating features using information from the test set: - Target encoding computed on full data - PCA components computed on full feature matrix - Feature selection based on full-data correlations **3. Temporal Leakage** Using future information in features: - Features computed from full time series (moving averages spanning train/test boundary) - External data joined without temporal awareness **4. Selection Leakage** Choosing the model or hyperparameters based on test performance, then 'evaluating' on the same test set.7f:Ta40,from sklearn.pipeline import Pipeline from sklearn.preprocessing import StandardScaler from sklearn.feature_selection import SelectKBest, f_classif from sklearn.linear_model import LogisticRegression from sklearn.model_selection import train_test_split # ============================================ # Leakage-Free Pipeline Pattern # ============================================ # The key insight: ALL preprocessing must be inside the pipeline # The pipeline is fitted only on training data # Split FIRST, before any processing X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Build pipeline with all preprocessing steps pipeline = Pipeline([ ('scaler', StandardScaler()), # Fitted on train only ('feature_selection', SelectKBest( # Fitted on train only score_func=f_classif, k=5 )), ('classifier', LogisticRegression()) # Fitted on train only ]) # Fit entire pipeline on training data only pipeline.fit(X_train, y_train) # Evaluate on test data - preprocessing uses train-fitted transforms test_score = pipeline.score(X_test, y_test) # ============================================ # Detecting Leakage: Sanity Checks # ============================================ def leakage_sanity_check(train_error, test_error, baseline): """ Warning signs that suggest potential leakage: 1. Test error much lower than expected for problem complexity 2. Test error surprisingly close to train error (low gap) 3. Performance doesn't degrade on truly new data """ gap = train_error - test_error warnings = [] if test_error < baseline * 0.5: warnings.append("Test error suspiciously low vs baseline") if abs(gap) < 0.01: # Near-perfect generalization warnings.append("Train-test gap suspiciously small") if gap < 0: # Test better than train warnings.append("CRITICAL: Test error lower than train error") return warnings # ============================================ # Temporal Leakage Prevention # ============================================ import pandas as pd def temporal_train_test_split(df, date_col, test_start_date): """ Split data temporally: train on past, test on future. No shuffling - respects temporal order. """ df = df.sort_values(date_col) train_mask = df[date_col] < test_start_date test_mask = df[date_col] >= test_start_date # Verify no overlap assert (train_mask & test_mask).sum() == 0, "Overlap detected!" return df[train_mask].copy(), df[test_mask].copy()80:T10db,import numpy as np import pandas as pd from scipy import stats from scipy.stats import ks_2samp, chi2_contingency import matplotlib.pyplot as plt def diagnose_split_quality(train_df, test_df, target_col=None): """ Comprehensive diagnostics for train-test split quality. Returns warnings if significant distribution differences detected. """ results = {'warnings': [], 'feature_analysis': {}} # Get common columns common_cols = set(train_df.columns) & set(test_df.columns) if target_col: common_cols.discard(target_col) for col in common_cols: train_data = train_df[col].dropna() test_data = test_df[col].dropna() if train_df[col].dtype in ['int64', 'float64']: # Continuous feature: KS test ks_stat, ks_pval = ks_2samp(train_data, test_data) results['feature_analysis'][col] = { 'type': 'continuous', 'ks_statistic': ks_stat, 'ks_pvalue': ks_pval, 'train_mean': train_data.mean(), 'test_mean': test_data.mean(), 'train_std': train_data.std(), 'test_std': test_data.std() } if ks_pval < 0.01: # Significant difference results['warnings'].append( f"Feature '{col}': Significant distribution difference " f"(KS p-value: {ks_pval:.4f})" ) else: # Categorical feature: Chi-square test train_counts = train_data.value_counts() test_counts = test_data.value_counts() # Align categories all_cats = set(train_counts.index) | set(test_counts.index) train_aligned = [train_counts.get(c, 0) for c in all_cats] test_aligned = [test_counts.get(c, 0) for c in all_cats] if len(all_cats) > 1 and min(train_aligned) > 0 and min(test_aligned) > 0: contingency = np.array([train_aligned, test_aligned]) chi2, chi2_pval, _, _ = chi2_contingency(contingency) results['feature_analysis'][col] = { 'type': 'categorical', 'chi2_statistic': chi2, 'chi2_pvalue': chi2_pval, 'n_categories': len(all_cats) } if chi2_pval < 0.01: results['warnings'].append( f"Feature '{col}': Significant distribution difference " f"(Chi2 p-value: {chi2_pval:.4f})" ) # Target analysis if target_col: train_target = train_df[target_col] test_target = test_df[target_col] if train_target.dtype in ['int64', 'float64'] and train_target.nunique() > 10: # Regression target results['target_analysis'] = { 'train_mean': train_target.mean(), 'test_mean': test_target.mean(), 'train_std': train_target.std(), 'test_std': test_target.std() } else: # Classification target train_dist = train_target.value_counts(normalize=True).to_dict() test_dist = test_target.value_counts(normalize=True).to_dict() results['target_analysis'] = { 'train_distribution': train_dist, 'test_distribution': test_dist } # Check for significant class proportion differences for cls in set(train_dist.keys()) | set(test_dist.keys()): train_prop = train_dist.get(cls, 0) test_prop = test_dist.get(cls, 0) if abs(train_prop - test_prop) > 0.05: results['warnings'].append( f"Target class '{cls}': Proportion differs by " f"{abs(train_prop - test_prop):.2%}" ) return results # Example usage results = diagnose_split_quality(df_train, df_test, target_col='target') print(f"Warnings found: {len(results['warnings'])}") for warning in results['warnings']: print(f" ⚠️ {warning}")72:{"json":{"id":"seed_9bb339ff9f3aa86e7eafa08db0fc466b","slug":"holdout-validation","category":"Machine Learning","topic":"Cross-Validation & Resampling","title":"Holdout Validation","description":"Master the foundational techniques for estimating model generalization performance through holdout validation methods. Learn train-test splits, validation sets, stratification strategies, and understand the fundamental limitations that motivate more advanced cross-validation approaches.","difficulty":"INTERMEDIATE","estimatedMinutes":90,"pages":[{"id":"seed_96f58974343fea0b99df7e45d06569f4","title":"Train-Test Split","order":0,"content":[{"id":"opening-hook","title":"The Fundamental Question of Generalization","content":[{"data":{"content":"Every machine learning model faces a single, defining question: **How will it perform on data it has never seen before?**\n\nThis isn't merely an academic concern—it's the very essence of why we build ML systems. A model that memorizes training data perfectly but fails on new data is worthless. A model that generalizes well to unseen examples is valuable. The entire discipline of machine learning revolves around this distinction between *memorization* and *generalization*.\n\nThe **train-test split** is our first and most fundamental tool for answering this question. It provides a simple yet powerful methodology: hide some data from the model during training, then reveal how well the model performs on this hidden data. This simple idea—so intuitive that it might seem obvious—forms the bedrock upon which all model evaluation rests."},"type":"text"},{"data":{"icon":"Target","title":"What You Will Learn","content":"By the end of this page, you will understand the theoretical foundations of train-test splitting, its mathematical justification, optimal split ratios for different scenarios, implementation best practices, and the subtle pitfalls that can invalidate your generalization estimates. You'll gain the precision and depth expected of a Principal ML Engineer.","variant":"info"},"type":"callout"}]},{"id":"generalization-problem","title":"The Generalization Problem","content":[{"data":{"content":"To understand why train-test splitting is essential, we must first deeply understand the **generalization problem** in machine learning. This problem arises from a fundamental tension:\n\n- We want our model to perform well on *future, unseen data* (the true objective)\n- We can only train and evaluate on *historical, observed data* (what we have access to)\n\nThis gap between what we want and what we can measure creates the core challenge. Let's formalize this precisely."},"type":"text"},{"data":{"content":"**The Statistical Learning Framework**\n\nIn the statistical learning framework, we assume:\n\n1. There exists an unknown joint probability distribution $P(X, Y)$ over features $X$ and targets $Y$\n2. Our dataset $\\mathcal{D} = \\{(x_1, y_1), \\ldots, (x_n, y_n)\\}$ consists of i.i.d. samples from this distribution\n3. We seek a function $f: X \\rightarrow Y$ that minimizes the **expected risk** (generalization error):\n\n$$R(f) = \\mathbb{E}_{(X,Y) \\sim P}[L(Y, f(X))]$$\n\nwhere $L$ is our loss function (e.g., squared error, cross-entropy).\n\nThe critical insight is that we cannot compute $R(f)$ directly—we don't know $P(X,Y)$. We can only compute the **empirical risk** on our finite sample:\n\n$$\\hat{R}_n(f) = \\frac{1}{n} \\sum_{i=1}^{n} L(y_i, f(x_i))$$"},"type":"text"},{"data":{"icon":"AlertCircle","title":"The Optimism of Training Error","content":"When we evaluate a model on the same data used for training, the empirical risk is systematically optimistic—it underestimates the true generalization error. This is because the model has adapted to the specific noise and idiosyncrasies in the training data. The more flexible the model, the more severe this optimism.","variant":"warning"},"type":"callout"},{"data":{"content":"**Quantifying the Optimism**\n\nThe gap between training error and generalization error is precisely what we need to estimate. Let $\\hat{f}$ be the model fitted on training data $\\mathcal{D}_{train}$. The **optimism** is defined as:\n\n$$\\text{Optimism} = R(\\hat{f}) - \\hat{R}_{train}(\\hat{f})$$\n\nFor many models, statistical theory provides bounds on this optimism. For instance, in linear regression with $p$ parameters, the expected optimism is approximately:\n\n$$\\mathbb{E}[\\text{Optimism}] \\approx \\frac{2p\\sigma^2}{n}$$\n\nwhere $\\sigma^2$ is the noise variance and $n$ is the sample size. This elegant result (related to Mallows' Cp and AIC) quantifies how training error systematically underestimates test error.\n\nHowever, these theoretical corrections only work for specific model families. For general models—especially complex ones like neural networks or gradient boosting—we need an empirical approach: the train-test split."},"type":"text"}],"titleType":"number","titleNumber":1},{"id":"train-test-methodology","title":"The Train-Test Split Methodology","content":[{"data":{"content":"$7a"},"type":"text"},{"data":{"icon":"Info","title":"The Principle of Independence","content":"The validity of test error as a generalization estimate rests on a crucial assumption: the test set must be statistically independent from the training process. Any leakage of test set information into training—even subtle leakage—invalidates the estimate and leads to over-optimistic performance assessments.","variant":"note"},"type":"callout"},{"data":{"content":"**Statistical Properties of Test Error**\n\nThe test error $\\hat{R}_{test}$ is a random variable (random because the test set is a random sample). Let's analyze its properties:\n\n**Unbiasedness:** Under the assumption that test data comes from the same distribution as future data:\n$$\\mathbb{E}[\\hat{R}_{test}] = R(\\hat{f})$$\n\nThe test error is an unbiased estimator of the generalization error *for the model trained on that particular training set*.\n\n**Variance:** The variance of the test error estimate is:\n$$\\text{Var}(\\hat{R}_{test}) = \\frac{\\text{Var}(L(Y, \\hat{f}(X)))}{n_{test}}$$\n\nThis tells us something critical: **larger test sets give more precise estimates**. The standard error of our estimate decreases with $1/\\sqrt{n_{test}}$.\n\n**Confidence Intervals:** For sufficiently large test sets, we can construct approximate confidence intervals:\n$$\\hat{R}_{test} \\pm z_{\\alpha/2} \\cdot \\sqrt{\\frac{\\hat{\\sigma}^2_{test}}{n_{test}}}$$\n\nwhere $\\hat{\\sigma}^2_{test}$ is the sample variance of losses on the test set."},"type":"text"}],"titleType":"number","titleNumber":2},{"id":"split-ratios","title":"Split Ratios and Tradeoffs","content":[{"data":{"content":"One of the most common questions practitioners face is: **What proportion of data should go into training versus testing?** The answer involves navigating a fundamental tradeoff that has profound implications for both model quality and evaluation reliability."},"type":"text"},{"data":{"content":"**The Bias-Variance Tradeoff in Splitting**\n\nThe split ratio creates a tension between two competing objectives:\n\n**More training data →** Better model (lower bias in performance)\n- The model sees more examples of the underlying pattern\n- Learning algorithms can extract more nuanced relationships\n- Especially important for high-capacity models with many parameters\n\n**More test data →** Better estimate (lower variance in evaluation)\n- Larger samples give more precise error estimates\n- Reduces uncertainty about true generalization performance\n- Enables more reliable model comparisons\n\nLet's quantify this tradeoff mathematically."},"type":"text"},{"data":{"content":"**Mathematical Analysis of the Tradeoff**\n\nSuppose our dataset has $n$ total samples. Let $\\alpha$ be the fraction used for testing, so:\n- $n_{test} = \\alpha n$\n- $n_{train} = (1-\\alpha) n$\n\nThe **estimation variance** of our test error is:\n$$\\text{Var}(\\hat{R}_{test}) \\propto \\frac{1}{\\alpha n}$$\n\nMeanwhile, the **model suboptimality** (gap between our model and what we could learn with all data) typically scales as:\n$$\\text{Suboptimality} \\propto \\frac{1}{(1-\\alpha) n}$$\n\n(with the exact exponent depending on the learning algorithm and problem complexity).\n\nThe total error in our evaluation can be decomposed as:\n$$\\text{Total Error} \\approx \\underbrace{\\text{Bias from smaller training set}}_{f((1-\\alpha)n)} + \\underbrace{\\text{Variance from smaller test set}}_{g(\\alpha n)}$$"},"type":"text"},{"data":{"rows":[["90/10","90%","10%","Large datasets (>100k samples)","Maximizes training data; sufficient test samples for precise estimates"],["80/20","80%","20%","Medium datasets (10k-100k samples)","Classical default; balances both objectives well"],["70/30","70%","30%","Smaller datasets (1k-10k samples)","Prioritizes evaluation reliability when data is limited"],["50/50","50%","50%","Very small datasets or model comparison studies","Often suboptimal; prefer cross-validation instead"]],"caption":"Common Train-Test Split Ratios by Context","headers":["Split Ratio","Training %","Test %","Best Use Case","Considerations"]},"type":"table"},{"data":{"icon":"Lightbulb","title":"Rule of Thumb for Test Set Size","content":"A useful heuristic: your test set should have at least 30 samples per class (for classification) or enough samples to give standard errors of ±2-3% on your primary metric. With fewer samples, confidence intervals become so wide that model comparisons lose statistical power.","variant":"tip"},"type":"callout"},{"data":{"content":"**Modern Perspective: The 'Enough Data' Threshold**\n\nIn the era of big data, the tradeoff looks different. With millions of samples:\n\n- Even 1% held out for testing might be 10,000+ samples—plenty for precise estimates\n- The marginal benefit of additional training samples diminishes (learning curves flatten)\n- The primary constraint becomes computational, not statistical\n\nFor deep learning on massive datasets, splits like 98/1/1 (train/validation/test) are common. The key insight: **once you have 'enough' test samples for your precision needs, additional test data doesn't help much, so put it in training**.\n\nConversely, with small datasets (<1,000 samples), even generous test sets have high variance. This is where cross-validation becomes essential—but that's a topic for the next module."},"type":"text"}],"titleType":"number","titleNumber":3},{"id":"implementation-best-practices","title":"Implementation Best Practices","content":[{"data":{"content":"Implementing train-test splits correctly requires attention to several technical details. Mistakes here can lead to data leakage, biased estimates, or irreproducible results. Let's examine the gold-standard practices used in production ML systems."},"type":"text"},{"data":{"fileName":"train_test_split_examples","languages":[{"code":"$7b","label":"Python","value":"python"},{"code":"$7c","label":"R","value":"r"}]},"type":"code"},{"data":{"items":["**Always set random seeds** — Reproducibility is non-negotiable in production ML. Document the seed in code, configuration, and experiment tracking.","**Use stratification for classification** — Especially critical with imbalanced classes or small datasets. Ensures both splits reflect the true class distribution.","**Split before any preprocessing** — Never fit scalers, encoders, or any transformation on the full dataset. Fit only on training data, then transform test data.","**Preserve indices for auditing** — Store which samples went into which split. Essential for debugging and regulatory compliance.","**Never look at test data during development** — The test set should be truly held out until final evaluation. Multiple peeks invalidate the estimate."],"title":{"icon":"CheckCircle","text":"Critical Implementation Guidelines","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"}],"titleType":"number","titleNumber":4},{"id":"random-shuffling","title":"Random Shuffling and Its Implications","content":[{"data":{"content":"The seemingly simple decision to 'shuffle the data before splitting' carries profound implications. Understanding when to shuffle—and critically, when *not* to—separates competent practitioners from those who unknowingly corrupt their evaluation."},"type":"text"},{"data":{"content":"$7d"},"type":"text"},{"data":{"icon":"AlertCircle","title":"When NOT to Shuffle: Temporal Data","content":"For time series data or any data with temporal dependencies, random shuffling is DANGEROUS. It creates 'future leakage'—using future information to predict the past. Always use temporal splits for sequential data: train on the past, test on the future.","variant":"error"},"type":"callout"},{"data":{"left":[{"data":{"items":["Cross-sectional data (independent samples at one time point)","Image classification with independent images","Tabular data without temporal or group structure","Text classification with independent documents","When sample order is arbitrary/administrative"],"title":{"icon":"CheckCircle","text":"When to Shuffle","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"}],"right":[{"data":{"items":["Time series forecasting (any temporal prediction)","Financial data with temporal dependencies","Sequential user behavior prediction","When detecting distribution drift is important","When future data shouldn't inform past predictions"],"title":{"icon":"XCircle","text":"When NOT to Shuffle","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"accent"},"type":"list"}]},"type":"two-column"},{"data":{"content":"**Grouped Data: A Special Consideration**\n\nWhen data contains groups (e.g., multiple samples from same patient, multiple frames from same video, multiple transactions from same user), shuffling at the sample level creates another form of leakage:\n\n- Training and test might contain different samples from the *same group*\n- The model learns group-specific patterns that won't generalize\n- Test performance is artificially inflated\n\n**Solution:** Use group-aware splitting. Entire groups must go into either training or testing, never split across both. This is covered in detail in the module on Group Cross-Validation."},"type":"text"}],"titleType":"number","titleNumber":5},{"id":"data-leakage","title":"The Data Leakage Problem","content":[{"data":{"content":"Data leakage is the most insidious failure mode in machine learning evaluation. It occurs when information from the test set inappropriately influences the training process or model selection. The result: evaluation metrics that look excellent but don't reflect real-world performance."},"type":"text"},{"data":{"content":"$7e"},"type":"text"},{"data":{"icon":"AlertCircle","title":"The Magnitude of Leakage Impact","content":"Leakage doesn't just add noise—it systematically inflates performance. In Kaggle competitions, top solutions with data leakage have dropped from 1st place to 1000th when leakage was removed. In production, models with leakage often fail catastrophically on truly new data.","variant":"warning"},"type":"callout"},{"data":{"fileName":"preventing_leakage","languages":[{"code":"$7f","label":"Python","value":"python"}]},"type":"code"}],"titleType":"number","titleNumber":6},{"id":"evaluating-split-quality","title":"Evaluating Split Quality","content":[{"data":{"content":"A good train-test split should produce training and test sets that are *statistically similar* (representing the same underlying distribution). Several diagnostic techniques help verify split quality."},"type":"text"},{"data":{"content":"**Distribution Comparison Tests**\n\nFor each feature, compare distributions between train and test:\n\n**Continuous Features:**\n- Kolmogorov-Smirnov test: Tests if two samples come from same distribution\n- Wasserstein distance: Measures 'earth mover's distance' between distributions\n- Visual: Overlaid density plots or Q-Q plots\n\n**Categorical Features:**\n- Chi-square test: Compares frequency distributions\n- Proportion comparisons with confidence intervals\n- Visual: Side-by-side bar charts\n\n**Target Variable:**\n- Classification: Compare class proportions (should match if stratified)\n- Regression: Compare mean, median, variance, and distribution shape"},"type":"text"},{"data":{"fileName":"split_quality_diagnostics","languages":[{"code":"$80","label":"Python","value":"python"}]},"type":"code"},{"data":{"icon":"Lightbulb","title":"The Adversarial Validation Technique","content":"A powerful technique: train a classifier to distinguish train from test samples. If the classifier achieves AUC >> 0.5, the splits are too different. Features with high importance in this classifier indicate distribution mismatch. This technique also detects covariate shift between training and production data.","variant":"tip"},"type":"callout"}],"titleType":"number","titleNumber":7},{"id":"summary-key-takeaways","title":"Summary and Key Takeaways","content":[{"data":{"content":"The train-test split is foundational to honest model evaluation. Let's consolidate the essential principles:"},"type":"text"},{"data":{"type":"ordered","items":["**Test error estimates generalization** — The test set serves as a proxy for future unseen data, providing an unbiased estimate of real-world performance.","**Split ratios involve tradeoffs** — More training data improves the model; more test data improves the estimate. Balance based on dataset size and precision needs.","**Random shuffling breaks collection artifacts** — Shuffle i.i.d. data to ensure representative splits, but never shuffle temporal or grouped data.","**Leakage invalidates everything** — Any information flow from test to training corrupts your estimate. Use pipelines and strict separation.","**Reproducibility is mandatory** — Always set random seeds and document split procedures. Irreproducible experiments are unscientific.","**Verify split quality** — Use statistical tests to confirm train and test distributions match. Catch problems before they corrupt downstream analysis."],"title":{"icon":"CheckCircle","text":"Core Principles","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"content":"**The Bigger Picture**\n\nWhile the train-test split is fundamental, it has significant limitations:\n\n- **High variance**: A single split may not represent the full data variability\n- **Wasteful**: Test data can never be used for training\n- **No hyperparameter tuning**: Using test set for tuning invalidates the estimate\n\nThese limitations motivate the **validation set** (covered next) and **cross-validation** (covered in Module 2). The train-test split remains essential as the final, unbiased evaluation step—but it's just one piece of a complete model development workflow."},"type":"text"},{"data":{"icon":"CheckCircle","title":"Page Complete","content":"You now understand the train-test split at a depth suitable for production ML systems. Next, we'll explore the validation set—a critical addition that enables hyperparameter tuning without compromising test set integrity.","variant":"success"},"type":"callout"}],"titleType":"number","titleNumber":8}],"isLocked":false},{"id":"seed_79ce601fb6fb7e631a58b96376a8bebf","title":"Validation Set","order":1,"content":null,"isLocked":true},{"id":"seed_996447f4b323ec6f15878c037211c300","title":"Stratification","order":2,"content":null,"isLocked":true},{"id":"seed_9d1b394b638941f99e49558d9d436181","title":"Random Seeds","order":3,"content":null,"isLocked":true},{"id":"seed_6ce67226b5c4b9cccbbc93c6f529015b","title":"Limitations","order":4,"content":null,"isLocked":true}]}}