Grepping a huge file (80GB) any way to speed it up?

bash grep
zzapper picture zzapper · Dec 17, 2012 · Viewed 73.3k times · Source 
 grep -i -A 5 -B 5 'db_pd.Clients'  eightygigsfile.sql

This has been running for an hour on a fairly powerful linux server which is otherwise not overloaded. Any alternative to grep? Anything about my syntax that can be improved, (egrep,fgrep better?)

The file is actually in a directory which is shared with a mount to another server but the actual diskspace is local so that shouldn't make any difference?

the grep is grabbing up to 93% CPU

Answer

dogbane picture dogbane · Dec 17, 2012

Here are a few options:

1) Prefix your grep command with LC_ALL=C to use the C locale instead of UTF-8.

2) Use fgrep because you're searching for a fixed string, not a regular expression.

3) Remove the -i option, if you don't need it.

So your command becomes:

LC_ALL=C fgrep -A 5 -B 5 'db_pd.Clients' eightygigsfile.sql

It will also be faster if you copy your file to RAM disk.

Related questions

How to use sed/grep to extract text between two words?

I am trying to output a string that contains everything between two words of a string: input: "Here is a String" output: "is a" Using: sed -n '/Here/,/String/p' includes the endpoints, but I don't want to include …

Read More
How to check if a file contains a specific string using Bash

I want to check if a file contains a specific string or not in bash. I used this script, but it doesn't work: if [[ 'grep 'SomeString' $File' ]];then # Some Actions fi What's wrong in my code?

Read More
How can I pipe stderr, and not stdout?

I have a program that writes information to stdout and stderr, and I need to process the stderr with grep, leaving stdout aside. Using a temporary file, one could do it in two steps: command > /dev/null 2> temp.…

Read More