Decision Tree
Python · Machine Learning (Scikit-Learn)
Theory
A decision tree predicts by asking a sequence of yes/no questions about the features, like a flowchart: 'is credit <= 723.5?', then 'is income <= 5.85?', until it reaches a leaf that gives the prediction. During fit it tries every feature and threshold and picks the split that makes the groups purest, measured by Gini impurity (or entropy) for classification and squared error for regression. It repeats this on each branch.
Trees handle numbers and encoded categories, need no scaling and can be read as rules with export_text. Their weakness is overfitting: left unchecked, a tree keeps splitting until it memorises the training data. Limit it with max_depth, min_samples_leaf or min_samples_split, and check train against test scores. feature_importances_ shows which features drove the splits.
Why we use it
Trees capture non-linear patterns and interactions automatically and produce rules that business users can follow, such as a clear credit approval policy. They are also the building block of random forests and gradient boosting.
When to use it
Use a shallow tree when you need explainable rules or a quick non-linear baseline. For the best accuracy, prefer a random forest or gradient boosting, which average many trees. Always limit depth to avoid overfitting.
How it works
- All training rows at the root
- Try every feature and threshold
- Pick the split that most reduces Gini impurity
- Reached max_depth or node already pure? Yes: Make a leaf: majority class. No: Split this node again.
- Repeat for each branch
- Rules: follow questions down to a leaf
Syntax
from sklearn.tree import DecisionTreeClassifier, export_text
tree = DecisionTreeClassifier(max_depth=3, min_samples_leaf=5, random_state=0)
tree.fit(X_train, y_train)
print(export_text(tree, feature_names=list(X.columns)))Example
tree = DecisionTreeClassifier(max_depth=2, random_state=0).fit(X_train, y_train)
print(export_text(tree, feature_names=list(X.columns)))Key points
- Splits are chosen to make groups as pure as possible (Gini)
- No scaling needed; works on thresholds
- export_text prints the learned rules
- Deep trees overfit: limit max_depth or min_samples_leaf
- feature_importances_ shows which features were used most
- A split can lower impurity even if both leaves predict the same class