PCA (Dimensionality Reduction)
Python · Machine Learning (Scikit-Learn)
Theory
Principal Component Analysis (PCA) replaces many correlated columns with a few new ones called principal components. The first component is the direction in the data with the most variance, the second is the direction with the most remaining variance at right angles to the first, and so on. Each component is a weighted mix of the original columns (pca.components_), and explained_variance_ratio_ says what share of the total variance each one keeps.
If maths, physics and chemistry marks rise and fall together, and English and Hindi do too, five subjects collapse into about two components, roughly 'science' and 'language', keeping most of the information. Standardise first, because PCA is driven by variance and a column with big units would dominate. PCA(n_components=2) keeps two, and PCA(n_components=0.95) keeps as many as needed for 95% of the variance.
Why we use it
Fewer columns make data easier to plot (two components fit a scatter plot), speed up models, reduce noise and remove redundancy between correlated features, which can help models that struggle with many inputs.
When to use it
Use PCA to visualise high-dimensional data, compress many correlated numeric features or speed up distance-based models. Avoid it when you must explain predictions in terms of original columns, since components are mixtures; try feature selection instead.
How it works
- Many correlated numeric features
- Standardise every column
- PCA finds the direction of greatest variance (PC1)
- Next components: most remaining variance, at right angles
- Enough cumulative variance kept (e.g. 90-95%)? Yes: Keep those components. No: Add another component.
- Smaller feature matrix for plotting or modelling
Syntax
from sklearn.decomposition import PCA
X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)
pca.explained_variance_ratio_ pca.components_Example
pca = PCA(n_components=0.95)
X_reduced = pca.fit_transform(X_scaled)
print(pca.n_components_, pca.explained_variance_ratio_.round(3))Key points
- Components are new axes ordered by variance explained
- Standardise features before PCA
- explained_variance_ratio_ guides how many to keep
- n_components=0.95 keeps 95% of the variance
- Components mix columns, so they are harder to interpret