Remove duplicate lines without sorting

shell scripting filter uniq
Robottinosino picture Robottinosino · Jul 18, 2012 · Viewed 57.7k times · Source

I have a utility script in Python:

#!/usr/bin/env python
import sys
unique_lines = []
duplicate_lines = []
for line in sys.stdin:
  if line in unique_lines:
    duplicate_lines.append(line)
  else:
    unique_lines.append(line)
    sys.stdout.write(line)
# optionally do something with duplicate_lines

This simple functionality (uniq without needing to sort first, stable ordering) must be available as a simple UNIX utility, mustn't it? Maybe a combination of filters in a pipe?

Reason for asking: needing this functionality on a system on which I cannot execute Python from anywhere.

Answer

Michael Hoffman picture Michael Hoffman · Jul 18, 2012

The UNIX Bash Scripting blog suggests:

awk '!x[$0]++'

This command is telling awk which lines to print. The variable $0 holds the entire contents of a line and square brackets are array access. So, for each line of the file, the node of the array x is incremented and the line printed if the content of that node was not (!) previously set.

Related questions

How do I split a string on a delimiter in Bash?

I have this string stored in a variable: IN="[email protected];[email protected]" Now I would like to split the strings by ; delimiter so that I have: ADDR1="[email protected]" ADDR2="[email protected]" I don't necessarily …

Read More
How to mkdir only if a directory does not already exist?

I am writing a shell script to run under the KornShell (ksh) on AIX. I would like to use the mkdir command to create a directory. But the directory may already exist, in which case I do not want to …

Read More
How can I declare and use Boolean variables in a shell script?

I tried to declare a Boolean variable in a shell script using the following syntax: variable=$false variable=$true Is this correct? Also, if I wanted to update that variable would I use the same syntax? Finally, is the following …

Read More