How to filter in NaN (pandas)?

python pandas nan
Gerhard picture Gerhard · Jul 31, 2014 · Viewed 117.6k times · Source

I have a pandas dataframe (df), and I want to do something like:

newdf = df[(df.var1 == 'a') & (df.var2 == NaN)]

I've tried replacing NaN with np.NaN, or 'NaN' or 'nan' etc, but nothing evaluates to True. There's no pd.NaN.

I can use df.fillna(np.nan) before evaluating the above expression but that feels hackish and I wonder if it will interfere with other pandas operations that rely on being able to identify pandas-format NaN's later.

I get the feeling there should be an easy answer to this question, but somehow it has eluded me. Any advice is appreciated. Thank you.

Answer

Mark Whitfield picture Mark Whitfield · Jul 31, 2014

This doesn't work because NaN isn't equal to anything, including NaN. Use pd.isnull(df.var2) instead.

Related questions

How to drop rows of Pandas DataFrame whose value in a certain column is NaN

I have this DataFrame and want only the records whose EPS column is not NaN: >>> df STK_ID EPS cash STK_ID RPT_Date 601166 20111231 601166 NaN NaN 600036 20111231 600036 NaN 12 600016 20111231 600016 4.3 NaN 601009 20111231 601009 NaN NaN 601939 20111231 601939 2.5 NaN 000001 20111231 000001 NaN NaN ...i.e. something like …

Read More
How to replace NaN values by Zeroes in a column of a Pandas Dataframe?

I have a Pandas Dataframe as below: itm Date Amount 67 420 2012-09-30 00:00:00 65211 68 421 2012-09-09 00:00:00 29424 69 421 2012-09-16 00:00:00 29877 70 421 2012-09-23 00:00:00 30990 71 421 2012-09-30 00:00:00 61303 72 485 2012-09-09 00:00:00 71781 73 485 2012-09-16 00:00:00 NaN 74 485 2012-09-23 00:00:00 11072 75 485 2012-09-30 00:00:00 113702 76 489 2012-09-09 00:00:00 64731 77 489 2012-09-16 00:00:00 NaN When I try to apply a …

Read More
How to check if any value is NaN in a Pandas DataFrame

In Python Pandas, what's the best way to check whether a DataFrame has one (or more) NaN values? I know about the function pd.isnan, but this returns a DataFrame of booleans for each element. This post right here doesn't …

Read More