Outliers & Validation
Python · Data Cleaning
Theory
Validation checks that data follows the rules you expect; outlier detection finds values that are technically valid but suspiciously far from the rest. Simple rules are boolean masks: df[~df['amount'].between(0, 100000)] lists amounts outside a sensible range, df[df['email'].isna()] finds missing emails and df['order_id'].is_unique confirms a key.
The IQR rule is a standard outlier test that is not distorted by the outliers themselves. Compute q1 = s.quantile(0.25), q3 = s.quantile(0.75) and iqr = q3 - q1; values below q1 - 1.5*iqr or above q3 + 1.5*iqr are outliers. With amounts 950, 1000, 1100, 1200, 1250 and 98000, only 98000 is flagged. A z-score ((x - mean) / std) above 3 is another rule, but one huge value inflates the std and can hide itself in small samples. After finding outliers, decide: fix, cap with clip(), drop, or keep and report.
Why we use it
A single wrong value (an extra zero, a test order, a negative quantity) can wreck an average or a chart. Rule checks and outlier tests catch these early, before they reach a report or a model.
When to use it
Run rule checks after loading and after each join (keys unique, no unexpected nulls, values in range). Use the IQR rule for skewed business numbers like order amounts; never drop outliers blindly - big customers are often real.
How it works
- Numeric column to validate
- Check hard rules: ranges, not null, unique keys
- Compute q1, q3 and iqr = q3 - q1
- Outside q1 - 1.5*iqr .. q3 + 1.5*iqr? Yes: Flag as outlier, investigate. No: Keep the row.
- Fix, cap, drop or keep each flagged value
- Validated data
Syntax
s.between(low, high)
q1, q3 = s.quantile([0.25, 0.75])
iqr = q3 - q1
df[s.between(q1 - 1.5*iqr, q3 + 1.5*iqr)]
df['key'].is_uniqueExample
q1, q3 = df['amount'].quantile([0.25, 0.75])
iqr = q3 - q1
clean = df[df['amount'].between(q1 - 1.5*iqr, q3 + 1.5*iqr)]Key points
- between(low, high) is inclusive on both ends by default
- IQR bounds: q1 - 1.5*IQR and q3 + 1.5*IQR
- Z-scores can miss outliers in small samples because the outlier inflates std
- Decide per case: fix, cap with clip(), drop, or keep and flag
- SQL equivalent: WHERE amount BETWEEN low AND high; PERCENTILE_CONT(0.25) for q1