Overfitting vs Underfitting
Python · Machine Learning (Scikit-Learn)
Theory
An underfit model is too simple to capture the pattern, so it scores poorly on both training and test data (high bias): a depth-1 tree that asks only one question is an example. An overfit model is so flexible that it memorises noise in the training data, scoring near-perfect on training but clearly worse on test data (high variance): an unlimited-depth tree is the classic case. The goal is the middle, where the test score is highest and close to the training score.
The practical check is to compare train and test (or cross-validated) scores as complexity grows. A big gap means overfitting: simplify the model (smaller max_depth, larger min_samples_leaf, stronger regularisation such as a lower C), add more data or use an ensemble. Both scores low means underfitting: use a more flexible model or better features.
Why we use it
Generalising to new data is the whole point of a model. Recognising the two failure modes from train and test scores tells you whether to make the model simpler or more powerful, instead of guessing.
When to use it
Check train vs test scores every time you fit a model, and plot them across a complexity setting (depth, k, C) when tuning. Use cross-validation for the comparison when data is small, so one lucky split does not mislead you.
How it works
- Fit the model; compute train and test scores
- Are both scores low? Yes: Underfitting: more complex model or better features. No: Continue.
- Train much higher than test? Yes: Overfitting: simplify, regularise or add data. No: Good fit.
- Repeat across complexity settings (depth, k, C)
- Choose the setting with the best test / CV score
Syntax
train_score = model.score(X_train, y_train)
test_score = model.score(X_test, y_test)
# gap = train_score - test_score
from sklearn.model_selection import validation_curveExample
for d in [1, 3, None]:
t = DecisionTreeClassifier(max_depth=d).fit(X_train, y_train)
print(d, t.score(X_train, y_train), t.score(X_test, y_test))Key points
- Underfit: low train and low test scores
- Overfit: high train score, much lower test score
- Aim for the highest test score with a small gap
- Fix overfitting: simplify, regularise, add data, use ensembles
- Fix underfitting: more flexible model or better features