bash, Linux: Set difference between two text files

bash file-io set-difference
Adam Matan picture Adam Matan · Mar 24, 2010 · Viewed 28.1k times · Source

I have two files A-nodes_to_delete and B-nodes_to_keep. Each file has a many lines with numeric ids.

I want to have the list of numeric ids that are in nodes_to_delete but NOT in nodes_to_keep, e.g. alt text .

Doing it within a PostgreSQL database is unreasonably slow. Any neat way to do it in bash using Linux CLI tools?

UPDATE: This would seem to be a Pythonic job, but the files are really, really large. I have solved some similar problems using uniq, sort and some set theory techniques. This was about two or three orders of magnitude faster than the database equivalents.

Answer

msw picture msw · Mar 24, 2010

The comm command does that.

Related questions

How do I tell if a regular file does not exist in Bash?

I've used the following script to see if a file exists: #!/bin/bash FILE=$1 if [ -f $FILE ]; then echo "File $FILE exists." else echo "File $FILE does not exist." fi What's the correct syntax to use if I only want …

Read More
How to redirect output to a file and stdout

In bash, calling foo would display any output from that command on the stdout. Calling foo > output would redirect any output from that command to the file specified (in this case 'output'). Is there a way to redirect output …

Read More
Split files using tar, gz, zip, or bzip2

I need to compress a large file of about 17-20 GB. I need to split it into several files of around 1GB per file. I searched for a solution via Google and found ways using split and cat commands. But …

Read More