More efficient way to find & tar millions of files

bash find tar
Stu Thompson picture Stu Thompson · Apr 23, 2010 · Viewed 26.2k times · Source

I've got a job running on my server at the command line prompt for a two days now:

find data/ -name filepattern-*2009* -exec tar uf 2009.tar {} ;

It is taking forever, and then some. Yes, there are millions of files in the target directory. (Each file is a measly 8 bytes in a well hashed directory structure.) But just running... 

find data/ -name filepattern-*2009* -print > filesOfInterest.txt

...takes only two hours or so. At the rate my job is running, it won't be finished for a couple of weeks.. That seems unreasonable. Is there a more efficient to do this? Maybe with a more complicated bash script?

A secondary questions is "why is my current approach so slow?"

Answer

Matthew Mott picture Matthew Mott · Apr 23, 2010

One option is to use cpio to generate a tar-format archive:

$ find data/ -name "filepattern-*2009*" | cpio -ov --format=ustar > 2009.tar

cpio works natively with a list of filenames from stdin, rather than a top-level directory, which makes it an ideal tool for this situation.

Related questions

ls command: how can I get a recursive full-path listing, one line per file?

How can I get ls to spit out a flat list of recursive one-per-line paths? For example, I just want a flat listing of files with their full paths: /home/dreftymac/. /home/dreftymac/foo.txt /home/dreftymac/bar.txt /home/…

Read More
How can I exclude all "permission denied" messages from "find"?

I need to hide all permission denied messages from: find . > files_and_folders I am experimenting when such message arises. I need to gather all folders and files, to which it does not arise. Is it possible to direct …

Read More
Find the files that have been changed in last 24 hours

E.g., a MySQL server is running on my Ubuntu machine. Some data has been changed during the last 24 hours. What (Linux) scripts can find the files that have been changed during the last 24 hours? Please list the file names, …

Read More