Random Forest
Python · Machine Learning (Scikit-Learn)
Theory
A random forest is an ensemble of many decision trees whose votes are combined. Each tree is trained on a bootstrap sample (rows drawn with replacement) and, at every split, considers only a random subset of features. The trees therefore make different mistakes, and the forest's majority vote (or average, for regression) is much more stable and accurate than any single tree.
The main settings are n_estimators (number of trees; more is steadier but slower), max_depth and min_samples_leaf (tree size) and max_features (features tried per split). Like trees, forests need no scaling. feature_importances_ gives the average impurity reduction each feature produced, a quick view of what drives predictions, though it can favour features with many distinct values. oob_score=True scores each tree on the rows it did not see, a free validation estimate.
Why we use it
Forests are among the most reliable out-of-the-box models for tabular data: accurate, resistant to overfitting compared with a single tree, robust to outliers and unscaled features, and they rank feature importance for free.
When to use it
Use a random forest as a strong default for classification or regression on tables, especially with non-linear effects. Choose a single tree when rules must be readable, linear models when coefficients must be explained, and gradient boosting when squeezing out the last bit of accuracy.
How it works
- Training data
- Draw a bootstrap sample of rows for each tree
- Grow each tree, trying a random subset of features at each split
- Repeat for n_estimators trees
- Classification? Yes: Majority vote of the trees. No: Average of the trees.
- Final prediction + feature importances
Syntax
from sklearn.ensemble import RandomForestClassifier
forest = RandomForestClassifier(n_estimators=200, max_depth=None,
min_samples_leaf=1, random_state=0)
forest.fit(X_train, y_train)
forest.feature_importances_Example
forest = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
print(forest.score(X_test, y_test))
print(forest.feature_importances_)Key points
- Many trees on bootstrap samples with random feature subsets
- Majority vote (classification) or average (regression)
- Usually beats a single tree and overfits less
- feature_importances_ ranks features; noise features score low
- Set random_state for reproducible results