KNN
Python · Machine Learning (Scikit-Learn)
Theory
K-nearest neighbours (KNN) classifies a new point by looking at the k training points closest to it and taking a majority vote of their labels (for regression, the average of their values). There is no real training step: fit simply stores the data, and all the work happens at prediction time, when distances to every stored point are computed, usually straight-line (Euclidean) distance.
k controls the trade-off: k=1 follows every noisy point and overfits, while a very large k blurs the classes together. Odd k avoids ties in two-class problems, and a good value is chosen with cross-validation. Because it relies on distances, features must be scaled, otherwise a column in rupees swamps one in years. kneighbors(X) returns the distances and positions of the neighbours, and predict_proba gives the share of neighbours in each class.
Why we use it
KNN is intuitive ('you are like your neighbours'), needs no assumptions about the shape of the data and can capture irregular boundaries. It is a good baseline and a natural fit for similarity-based tasks such as recommendations.
When to use it
Use KNN for small to medium datasets with a few scaled numeric features. Avoid it for very large datasets (slow predictions) or many features, where distances become less meaningful; tree ensembles or linear models scale better.
How it works
- New point to classify
- Compute its distance to every training point
- Sort and keep the k nearest
- Count the labels among those k neighbours
- Is there a tie? Yes: Odd k avoids it; sklearn takes the first class. No: Majority label wins.
- Predicted class (and share of votes)
Syntax
from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)
knn.predict(X_new) knn.predict_proba(X_new)
dist, idx = knn.kneighbors(X_new)Example
knn = make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=5))
knn.fit(X_train, y_train)
print(knn.predict(X_test[:3]))Key points
- Predicts from the k closest training points
- fit only stores data; prediction does the work
- Always scale features: distances depend on units
- Small k overfits, large k underfits; tune with CV
- Slow on large datasets because every prediction compares to all rows