Data Cleaning
Python · Data Cleaning
Theory
Data cleaning fixes problems in raw data before analysis, and missing values are the first problem to tackle. pandas marks them as NaN (numbers), None or NaT (dates); df.isna() gives True where a value is missing and df.isna().sum() counts them per column. You then either fill or drop them.
fillna(0) fills every gap with one value; fillna({'city': 'Unknown', 'amount': 0}) uses a different value per column; fillna(df['amount'].median()) fills with a statistic. df.ffill() copies the last known value forward and df.bfill() copies the next one back - useful for time series and for 'carry the last price'. dropna() removes rows with any NaN, dropna(subset=['amount']) only checks some columns, dropna(thresh=2) keeps rows with at least 2 real values and dropna(how='all') drops rows that are completely empty. interpolate() estimates numeric gaps from neighbours, e.g. 100, NaN, 300 becomes 100, 200, 300.
Why we use it
Missing values quietly change results: mean() skips them, sum() treats them as 0, joins and groupbys may drop them, and many ML models refuse NaN. Handling them deliberately keeps totals and averages honest.
When to use it
Fill when a sensible default exists (0 orders, 'Unknown' city, last known price); drop when the row is useless without the value; interpolate for evenly spaced numeric readings. Always count NaN first so you know how much data a choice affects.
How it works
- Load raw data
- Count gaps with df.isna().sum()
- Is the row useless without this value? Yes: dropna(subset=[col]). No: Keep the row and fill.
- Sensible default exists? Yes: fillna(value), ffill() or interpolate(). No: Leave as NaN.
- Re-check isna().sum()
- DataFrame with missing values handled
Syntax
df.isna().sum()
df.fillna(value | {'col': value})
df.ffill(); df.bfill()
df.dropna(subset=[...], thresh=n, how='any'|'all')
s.interpolate()Example
df = df.fillna({'city': 'Unknown', 'amount': 0})
df = df.dropna(subset=['name'])Key points
- isna().sum() is the first check on any new dataset
- fillna with a dict fills each column with its own value
- Use df.ffill() / df.bfill(); fillna(method='ffill') is deprecated
- dropna(subset=[...]) and thresh= give finer control than dropping any NaN row
- SQL equivalent: COALESCE(city, 'Unknown') = fillna; WHERE amount IS NOT NULL = dropna(subset)
- Methods return new DataFrames - assign the result back