Overfitting, Underfitting & Train/Test Split

What it is

A model underfits when it never learned the pattern: training error is high and test error is high too. A straight line through curved data is the classic case. A model overfits when it learned the training data too well, including its noise: training error is tiny but error on new data is much worse, because it memorised examples instead of learning the rule.

The train/test split is how you catch both. Before training, you hold back a slice of the data, commonly 20 percent, that the model never sees. Training accuracy tells you whether it can learn at all; test accuracy tells you whether what it learned transfers.

Most real projects add a third slice, the validation set, used for choosing hyperparameters such as tree depth or learning rate. The test set is then touched once, at the very end, so its score is an honest estimate of production performance.

Why it's needed

This is the most common way a model that looked great in a notebook fails in production. A tree that scored 100 percent on training data and 70 percent on new customers is worse than a simpler model that scored 85 and 84. The gap between the two numbers is the warning sign, and without a held-out set you cannot see it.

It also decides what to do next. Underfitting calls for a more capable model or better features. Overfitting calls for more data, regularisation or a simpler model. Picking the wrong fix wastes days.

How it works

• Split before you do anything else: train_test_split(X, y, test_size=0.2). Fit scalers and encoders on the training slice only, then apply them to the test slice.
• Compare the two scores. High train, low test: overfitting. Low train, low test: underfitting. Both high and close together: healthy.
• Fix overfitting: more data, a simpler model, regularisation (L1/L2), dropout in neural networks, early stopping, or a lower max_depth for trees.
• Fix underfitting: a more flexible model, more informative features, longer training, or a higher learning rate if training stalled.
• Use k-fold cross-validation on small datasets so one lucky or unlucky split does not mislead you.
• Keep the test set for the end. Every time you tune against it, it stops being an honest measure.

Code example

import numpy as np
from sklearn.model_selection import train_test_split

# Noisy curved data: y = sin(x) + noise
rng = np.random.default_rng(0)
x = np.sort(rng.uniform(0, 6, 40))
y = np.sin(x) + rng.normal(0, 0.3, size=x.size)

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.25, random_state=0)

def mse(a, b):
    return np.mean((a - b) ** 2)

# Fit polynomials of increasing degree:
# degree 1 underfits, degree 3 is about right, degree 15 overfits
for degree in (1, 3, 15):
    coeffs = np.polyfit(x_train, y_train, degree)
    train_err = mse(np.polyval(coeffs, x_train), y_train)
    test_err = mse(np.polyval(coeffs, x_test), y_test)
    print(f"degree {degree:2d}: train MSE {train_err:.3f}   test MSE {test_err:.3f}")

Use cases

• Validating any model before deployment
• Diagnosing why a model performs well in testing but poorly in production
• Choosing between a simple and a complex model with evidence instead of instinct
• Deciding whether to collect more data or add regularisation

Common mistakes

• Scaling or encoding the whole dataset before splitting, which leaks test statistics into training.
• Tuning hyperparameters against the test set until it looks good, then reporting that number.
• Splitting time-series data randomly, so the model trains on the future and is tested on the past.
• Letting duplicate or near-duplicate rows land on both sides of the split.
• Chasing 100 percent training accuracy as if it were the goal.

Interview questions