How to merge sorted files without using a temporary file?

unix sorting command-line temporary-files
Matei David picture Matei David · Jul 6, 2011 · Viewed 8.3k times · Source

I'm trying to merge many sorted files in a UNIX/Linux script with sort -m, and I noticed that sort first writes the result to a temporary file, then copies it to destination. My understanding of -m was that it assumes the files are sorted, so using a temporary file is completely unnecessary, and it wastes both hard disk space and CPU cycles (I'm using sort in a pipeline which gets stuck waiting for sort to output anything.) Is there a way to tell sort to not use temporary files when merging sorted files? Or a better version which doesn't?

The exact CL looks like:

$ sort -m -s -t '_' -k 1,1n -k 2,2n <(gunzip <file_1) [...] <(gunzip <file_n) | gzip >output

I'm using sort from GNU coreutils 5.97.

Answer

Marcin picture Marcin · Jul 6, 2011

Check out these options from man sort, they might let you minimize the amount of space needed for merging.

--batch-size=NMERGE

merge at most NMERGE inputs at once; for more use temp files

--compress-program=PROG

compress temporaries with PROG; decompress them with PROG -d

Related questions

Is there a way to ignore header lines in a UNIX sort?

I have a fixed-width-field file which I'm trying to sort using the UNIX (Cygwin, in my case) sort utility. The problem is there is a two-line header at the top of the file which is being sorted to the bottom …

Read More
Sorting a tab delimited file

I have a data with the following format: foo<tab>1.00<space>1.33<space>2.00<tab>3 Now I tried to sort the file based on the last field decreasingly. I tried the following commands but …

Read More
Sorting multiple keys with Unix sort

I have potentially large files that need to be sorted by 1-n keys. Some of these keys might be numeric and some of them might not be. This is a fixed-width columnar file so there are no delimiters. Is there …

Read More