Python: Uniqueness for list of lists

python unique
Hellnar picture Hellnar · Sep 16, 2010 · Viewed 60.7k times · Source

I am curious what would be an efficient way of uniquefying such data objects:

testdata =[ ['9034968', 'ETH'], ['14160113', 'ETH'], ['9034968', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15724032', 'ETH'], ['15481740', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['10307528', 'ETH'], ['15481757', 'ETH'], ['15481724', 'ETH'], ['15481740', 'ETH'], ['15379365', 'ETH'], ['11111', 'NOT'], ['9555269', 'NOT'], ['15379365', 'ETH']
]

For each data pair, left numeric string PLUS the type at the right tells the uniqueness of a data element. The return value should be a list of lists as same as the testdata, but with only the unique values kept.

Answer

Mark Byers picture Mark Byers · Sep 16, 2010

You can use a set:

unique_data = [list(x) for x in set(tuple(x) for x in testdata)]

You can also see this page which benchmarks a variety of methods that either preserve or don't preserve order.

Related questions

Count unique values with pandas per groups

I need to count unique ID values in every domain I have data ID, domain 123, 'vk.com' 123, 'vk.com' 123, 'twitter.com' 456, 'vk.com' 456, 'facebook.com' 456, 'vk.com' 456, 'google.com' 789, 'twitter.com' 789, 'vk.com' I try df.groupby(['domain', 'ID']).count() …

Read More
How do you remove duplicates from a list whilst preserving order?

Is there a built-in that removes duplicates from list in Python, whilst preserving order? I know that I can use a set to remove duplicates, but that destroys the original order. I also know that I can roll my own …

Read More
Generate 'n' unique random numbers within a range

I know how to generate a random number within a range in Python. random.randint(numLow, numHigh) And I know I can put this in a loop to generate n amount of these numbers for x in range (0, n): listOfNumbers.…

Read More