Hyperparameter Tuning
Python · Machine Learning (Scikit-Learn)
Theory
Hyperparameters are settings you choose before training, such as max_depth of a tree, n_neighbors in KNN or C in logistic regression; the model does not learn them from data. GridSearchCV tries every combination in a param_grid, scores each one with cross-validation on the training set and keeps the best. After fit, best_params_ holds the winning settings, best_score_ its mean cross-validated score, and best_estimator_ a model refitted on the whole training set with those settings.
cv_results_ contains the full table of mean and std scores per combination, which is easy to view with pd.DataFrame. The number of fits grows fast: 3 values x 4 values x 5 folds is 60 fits, so RandomizedSearchCV, which samples n_iter combinations, is better for large grids. To tune a pipeline, name parameters as step__param, for example {'model__C': [0.1, 1, 10]}. Report the final score on the untouched test set.
Why we use it
Default settings are rarely the best for your data. Systematic, cross-validated search finds better settings without peeking at the test set, so the improvement you report is real rather than tuned to one lucky split.
When to use it
Tune after you have a working pipeline and a baseline score. Use GridSearchCV for a few parameters with a handful of values each, RandomizedSearchCV for many parameters or wide ranges. Keep the test set out of the search.
How it works
- Pipeline and a baseline score
- Define param_grid with a few values per setting
- For each combination run k-fold cross-validation
- Rank combinations by mean CV score
- Is the best value at the edge of the grid? Yes: Extend the grid and search again. No: Accept best_params_.
- Score best_estimator_ once on the test set
Syntax
from sklearn.model_selection import GridSearchCV
grid = GridSearchCV(estimator, param_grid={'max_depth': [2, 3, 5]}, cv=5, scoring='accuracy')
grid.fit(X_train, y_train)
grid.best_params_, grid.best_score_
grid.best_estimator_.score(X_test, y_test)Example
grid = GridSearchCV(pipe, {'model__C': [0.1, 1, 10]}, cv=5)
grid.fit(X_train, y_train)
print(grid.best_params_, grid.best_score_)Key points
- Hyperparameters are set before training, not learned
- GridSearchCV tries every combination with cross-validation
- best_estimator_ is refitted on all training data
- Grids explode in size; use RandomizedSearchCV for many options
- Pipeline parameters use step__param names