Data Filtering
Python · Pandas Basics
Theory
Filtering keeps only the rows that meet a condition, like SQL's WHERE. It works in two steps: df['amount'] > 1000 produces a boolean Series (True or False for every row), and df[mask] keeps the rows where the mask is True, with their original index labels preserved.
Combine conditions with & (and), | (or) and ~ (not), wrapping each condition in parentheses because of operator precedence: df[(df['city'] == 'Pune') & (df['amount'] > 1000)]. Python's and/or do not work here and raise an error. Helpful shortcuts are isin() for lists (df[df['city'].isin(['Pune', 'Delhi'])]), between() for inclusive ranges, str.contains() for text, isna()/notna() for missing values and df.query("city == 'Pune' and amount > 1000") for a SQL-like string. Use df.loc[mask, 'name'] to filter rows and pick columns together.
Why we use it
Analysis questions usually concern a subset - high-value orders, one region, last month's sales. Boolean filtering pulls that subset out in one readable line with no loops.
When to use it
Use boolean masks for row filters on any condition, and query() when a long condition reads better as text. To modify the filtered rows, use df.loc[mask, 'col'] = value rather than chained indexing, which can trigger SettingWithCopyWarning.
How it works
- df[(df['city'] == 'Pune') & (df['amount'] > 1000)]
- Evaluate each condition for every row
- Combine the boolean Series with & / | / ~
- Is the combined mask True for this row? Yes: Keep the row. No: Drop the row.
- New DataFrame of matching rows
Syntax
df[df['col'] > value]
df[(cond1) & (cond2)]
df[df['col'].isin([...])]
df.query("col > value")Example
big = df[(df['city'] == 'Pune') & (df['amount'] > 1000)]
print(big)Key points
- Combine conditions with & / | / ~, each wrapped in parentheses
- Returns a new DataFrame - the original is left unchanged
- isin() is SQL IN, ~isin() is NOT IN, between() is BETWEEN (inclusive)
- df.query("amount > @limit") reads like SQL and can use Python variables with @
- Use .loc[mask, col] = value to update filtered rows