Fastest possible grep

bash unix grep
pistacchio picture pistacchio · Jan 30, 2012 · Viewed 75.4k times · Source

I'd like to know if there is any tip to make grep as fast as possible. I have a rather large base of text files to search in the quickest possible way. I've made them all lowercase, so that I could get rid of -i option. This makes the search much faster.

Also, I've found out that -F and -P modes are quicker than the default one. I use the former when the search string is not a regular expression (just plain text), the latter if regex is involved.

Does anyone have any experience in speeding up grep? Maybe compile it from scratch with some particular flag (I'm on Linux CentOS), organize the files in a certain fashion or maybe make the search parallel in some way?

Answer

Chewie picture Chewie · Jan 30, 2012

Try with GNU parallel, which includes an example of how to use it with grep:

grep -r greps recursively through directories. On multicore CPUs GNU parallel can often speed this up.

find . -type f | parallel -k -j150% -n 1000 -m grep -H -n STRING {}

This will run 1.5 job per core, and give 1000 arguments to grep.

For big files, it can split it the input in several chunks with the --pipe and --block arguments:

 parallel --pipe --block 2M grep foo < bigfile

You could also run it on several different machines through SSH (ssh-agent needed to avoid passwords):

parallel --pipe --sshlogin server.example.com,server2.example.net grep foo < bigfile

Related questions

How can I format my grep output to show line numbers at the end of the line, and also the hit count?

I'm using grep to match string in a file. Here is an example file: example one, example two null, example three, example four null, grep -i null myfile.txt returns example two null, example four null, How can I return …

Read More
How do I use grep to search the current directory for all files having the a string "hello" yet display only .h and .cc files?

How do I use grep to search the current directory for any and all files containing the string "hello" and display only .h and .cc files?

Read More
How to concatenate multiple lines of output to one line?

If I run the command cat file | grep pattern, I get many lines of output. How do you concatenate all lines into one line, effectively replacing each "\n" with "\" " (end with " followed by space)? cat file | grep pattern | xargs sed …

Read More