Model Evaluation
Python · Machine Learning (Scikit-Learn)
Theory
Classification metrics start from the confusion matrix, which counts true positives (TP), true negatives (TN), false positives (FP, false alarms) and false negatives (FN, misses). Accuracy = correct / total. Precision = TP / (TP + FP): of the rows flagged positive, how many really were. Recall = TP / (TP + FN): of the real positives, how many were caught. F1 is the harmonic mean of precision and recall, high only when both are.
These all depend on the 0.5 threshold. ROC AUC uses the predicted probabilities instead and measures how well the model ranks positives above negatives across every threshold: 1 is perfect ranking, 0.5 is random guessing. In the diagram, one positive scored 0.4 (a miss) and one negative scored 0.6 (a false alarm), so accuracy, precision and recall are each 0.67, while AUC is 0.89 because most pairs are still ranked correctly.
Why we use it
A single accuracy number hides which mistakes a model makes. On imbalanced data, a model that always predicts 'not fraud' can be 99% accurate and useless; precision, recall, F1 and AUC reveal that.
When to use it
Use precision when false alarms are expensive (spam filters), recall when misses are expensive (fraud, disease), F1 to balance both, and ROC AUC to compare models independent of threshold. Accuracy is fine only for balanced classes.
How it works
- Fitted classifier and test set
- Get predictions and probabilities for X_test
- Build the confusion matrix: TN, FP, FN, TP
- Are false alarms or misses more costly? Yes: Focus on precision (alarms) or recall (misses). No: Use F1 or accuracy.
- Compare models threshold-free with ROC AUC
- Choose the model and threshold
Syntax
from sklearn.metrics import (accuracy_score, precision_score, recall_score,
f1_score, confusion_matrix, roc_auc_score, classification_report)
confusion_matrix(y_test, y_pred)
roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])Example
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
print(roc_auc_score(y_test, prob))Key points
- Confusion matrix layout: [[TN, FP], [FN, TP]]
- Precision = TP / (TP + FP); recall = TP / (TP + FN)
- F1 balances precision and recall
- ROC AUC needs probabilities, not 0/1 predictions
- Accuracy misleads on imbalanced classes