Pipelines and ColumnTransformer
Python · Machine Learning (Scikit-Learn)
Theory
A Pipeline chains preprocessing steps and a final model into one estimator: Pipeline([('scale', StandardScaler()), ('model', LogisticRegression())]). Calling fit runs fit_transform on each step in order and fits the model; calling predict runs transform on each step and predicts. make_pipeline does the same and names the steps automatically.
Real tables need different treatment per column, which is what ColumnTransformer does: numeric columns go through an imputer and scaler, categorical ones through an imputer and OneHotEncoder, and the outputs are joined side by side. Put the ColumnTransformer inside a Pipeline with the model and the whole thing fits on a raw DataFrame. Because every step is fitted only on the data passed to fit, cross-validation and GridSearchCV with a pipeline cannot leak test information. Step parameters are addressed as step__param, for example model__C.
Why we use it
Without a pipeline it is easy to fit a scaler on all the data, forget a step at prediction time or apply steps in the wrong order. A pipeline packages the full recipe so training, evaluation and deployment use exactly the same transformations.
When to use it
Use a pipeline whenever there is any preprocessing before the model, and always when cross-validating or tuning. Use ColumnTransformer when numeric and categorical columns need different steps. Save the whole pipeline, not just the model.
How it works
- Raw DataFrame: numeric + categorical columns
- ColumnTransformer splits columns by type
- Numeric: impute median, then scale
- Categorical: impute most frequent, then one-hot
- Outputs joined side by side into one matrix
- Final step: the model fits / predicts
- One estimator: pipe.fit(X, y), pipe.predict(X_new)
Syntax
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.compose import ColumnTransformer
pre = ColumnTransformer([('num', num_pipe, num_cols), ('cat', cat_pipe, cat_cols)])
clf = Pipeline([('prep', pre), ('model', LogisticRegression())])
clf.fit(X_train, y_train)Example
pre = ColumnTransformer([('num', StandardScaler(), ['income']),
('cat', OneHotEncoder(handle_unknown='ignore'), ['city'])])
clf = make_pipeline(pre, LogisticRegression())
clf.fit(X_train, y_train)Key points
- fit runs fit_transform on each step, then fits the model
- ColumnTransformer sends each column group to its own steps
- Pipelines prevent leakage in cross-validation and tuning
- Refer to step parameters as step__param, e.g. model__C
- Save and deploy the whole pipeline, not just the model