I'd like to concatenate two dataframes A, B to a new one without duplicate rows (if rows in B already exist in A, don't add):
Dataframe A: Dataframe B:
I II I II
0 1 2 5 6
1 3 1 3 1
New Dataframe:
I II
0 1 2
1 3 1
2 5 6
How can I do this?
The simplest way is to just do the concatenation, and then drop duplicates.
>>> df1
A B
0 1 2
1 3 1
>>> df2
A B
0 5 6
1 3 1
>>> pandas.concat([df1,df2]).drop_duplicates().reset_index(drop=True)
A B
0 1 2
1 3 1
2 5 6
The reset_index(drop=True)
is to fix up the index after the concat()
and drop_duplicates()
. Without it you will have an index of [0,1,0]
instead of [0,1,2]
. This could cause problems for further operations on this dataframe
down the road if it isn't reset right away.