How to compare two huge text files (more than 50gb each)?

bash unix text-files diff file-comparison
sara picture sara · Apr 12, 2016 · Viewed 15.1k times · Source

I have 2 huge text files and want to find the difference between them? What is the fast command/ utility/ or script to do this job?

I try using unix diff but, it failed for huge files. It show me "Permission Denied"

Also, I try unix bdiff (as I read that is good for huge files) but, I did not complete test it because it show me "bdiff: command not found"

Also, I try windows FC (file compare) but, I did not know how to show the output in a new text file. The result output in the cmd and I can't figure it all.

Any suggestion will help me a lot. Please help me in this matter.

Thanks a lot.

Answer

SLePort picture SLePort · Apr 12, 2016

You can try diff with the --speed-large-files option :

diff --speed-large-files file1 file2

cmp is another alternative (compare files byte by byte) :

cmp file1 file2

Related questions

How can I check if a directory exists in a Bash shell script?

What command can be used to check if a directory exists or not, within a Bash shell script?

Read More
Looping through the content of a file in Bash

How do I iterate through each line of a text file with Bash? With this script: echo "Start!" for p in (peptides.txt) do echo "${p}" done I get this output on the screen: Start! ./runPep.sh: line 3: syntax error …

Read More
How to permanently set $PATH on Linux/Unix?

I'm trying to add a directory to my path so it will always be in my Linux path. I've tried: export PATH=$PATH:/path/to/dir This works, however each time I exit the terminal and start a new terminal …

Read More