I have a dataframe with ~300K rows and ~40 columns. I want to find out if any rows contain null values - and put these 'null'-rows into a separate dataframe so that I could explore them easily.
I can create a mask explicitly:
mask = False
for col in df.columns:
mask = mask | df[col].isnull()
dfnulls = df[mask]
Or I can do something like:
df.ix[df.index[(df.T == np.nan).sum() > 1]]
Is there a more elegant way of doing it (locating rows with nulls in them)?
[Updated to adapt to modern pandas
, which has isnull
as a method of DataFrame
s..]
You can use isnull
and any
to build a boolean Series and use that to index into your frame:
>>> df = pd.DataFrame([range(3), [0, np.NaN, 0], [0, 0, np.NaN], range(3), range(3)])
>>> df.isnull()
0 1 2
0 False False False
1 False True False
2 False False True
3 False False False
4 False False False
>>> df.isnull().any(axis=1)
0 False
1 True
2 True
3 False
4 False
dtype: bool
>>> df[df.isnull().any(axis=1)]
0 1 2
1 0 NaN 0
2 0 0 NaN
[For older pandas
:]
You could use the function isnull
instead of the method:
In [56]: df = pd.DataFrame([range(3), [0, np.NaN, 0], [0, 0, np.NaN], range(3), range(3)])
In [57]: df
Out[57]:
0 1 2
0 0 1 2
1 0 NaN 0
2 0 0 NaN
3 0 1 2
4 0 1 2
In [58]: pd.isnull(df)
Out[58]:
0 1 2
0 False False False
1 False True False
2 False False True
3 False False False
4 False False False
In [59]: pd.isnull(df).any(axis=1)
Out[59]:
0 False
1 True
2 True
3 False
4 False
leading to the rather compact:
In [60]: df[pd.isnull(df).any(axis=1)]
Out[60]:
0 1 2
1 0 NaN 0
2 0 0 NaN