sorting large text data

python sorting bigdata
fodon picture fodon · Aug 16, 2011 · Viewed 11.3k times · Source

I have a large file (100 million lines of tab separated values - about 1.5GB in size). What is the fastest known way to sort this based on one of the fields?

I have tried hive. I would like to see if this can be done faster using python.

Answer

urschrei picture urschrei · Aug 16, 2011

Have you considered using the *nix sort program? in raw terms, it'll probably be faster than most Python scripts.

Use -t $'\t' to specify that it's tab-separated, -k n to specify the field, where n is the field number, and -o outputfile if you want to output the result to a new file. Example:

sort -t $'\t' -k 4 -o sorted.txt input.txt

Will sort input.txt on its 4th field, and output the result to sorted.txt

Related questions

How do I sort a dictionary by value?

I have a dictionary of values read from two fields in a database: a string field and a numeric field. The string field is unique, so that is the key of the dictionary. I can sort on the keys, but …

Read More
How can I sort a dictionary by key?

What would be a nice way to go from {2:3, 1:89, 4:5, 3:0} to {1:89, 2:3, 3:0, 4:5}? I checked some posts but they all use the "sorted" operator that returns tuples.

Read More
How do I sort a list of dictionaries by a value of the dictionary?

I have a list of dictionaries and want each item to be sorted by a specific value. Take into consideration the list: [{'name':'Homer', 'age':39}, {'name':'Bart', 'age':10}] When sorted by name, it should become: [{'name':'Bart', 'age':10}, {'name':'Homer', …

Read More