How to delete duplicate lines in a file without sorting it in Unix?

unix shell scripting sed awk
Vijay picture Vijay · Sep 18, 2009 · Viewed 106.4k times · Source

Is there a way to delete duplicate lines in a file in Unix?

I can do it with sort -u and uniq commands, but I want to use sed or awk. Is that possible?

Answer

Jonas Elfström picture Jonas Elfström · Sep 18, 2009 
awk '!seen[$0]++' file.txt

seen is an associative-array that Awk will pass every line of the file to. If a line isn't in the array then seen[$0] will evaluate to false. The ! is the logical NOT operator and will invert the false to true. Awk will print the lines where the expression evaluates to true. The ++ increments seen so that seen[$0] == 1 after the first time a line is found and then seen[$0] == 2, and so on.
Awk evaluates everything but 0 and "" (empty string) to true. If a duplicate line is placed in seen then !seen[$0] will evaluate to false and the line will not be written to the output.

Related questions

Remove part of path on Unix

I'm trying to remove part of the path in a string. I have the path: /path/to/file/drive/file/path/ I want to remove the first part /path/to/file/drive and produce the output: file/path/ Note: I …

Read More
unix - count of columns in file

Given a file with data like this (i.e. stores.dat file) sid|storeNo|latitude|longitude 2|1|-28.03720000|153.42921670 9|2|-33.85090000|151.03274200 What would be a command to output the number of column names? i.e. In the example above it would be 4. (number …

Read More
Unix - create path of folders and file

I know you can do mkdir to create a directory and touch to create a file, but is there no way to do both operations in one go? i.e. if I want to do the below when the folder …

Read More