How to delete duplicate lines in a file...AWK, SED, UNIQ not working on my file

sed awk duplicates lines uniq
Corepuncher picture Corepuncher · Sep 26, 2013 · Viewed 12.9k times · Source

I find many ways to do this, AWK, SED, UNIQ, but none of them are working on my file.

I want to delete duplicate lines. Here is an example of part of my file:

KTBX
KFSO
KCLK
KTBX
KFSO
KCLK
PAJZ
PAJZ

NOTE: I had to manually add line feeds when I cut and pasted from the file...for some reason it was putting all the variables on one line. Makes me think that my 44,000 line text file actually has only "1" line? Is there a way to modify it so I can delete dups?

Answer

philshem picture philshem · Sep 27, 2013

You can see all non-printed characters with this command:

od -c oldfile

If all your records are on one line, you can use sed to replace a whitespace (space, tab, newline) with a linebreak:

sed -e 's/\s\+/\n/g' oldfile > oldfile.1

Once you have multiple lines, this awk one-liner:

awk '!x[$0]++' oldfile.1 > newfile

my outfile:

KTBX
KFSO
KCLK
PAJZ

Related questions

How to do a recursive find/replace of a string with awk or sed?

How do I find and replace every occurrence of: subdomainA.example.com with subdomainB.example.com in every text file under the /home/www/ directory tree recursively?

Read More
Bash tool to get nth line from a file

Is there a "canonical" way of doing that? I've been using head -n | tail -1 which does the trick, but I've been wondering if there's a Bash tool that specifically extracts a line (or a range of lines) from a …

Read More
How to show only next line after the matched one?

grep -A1 'blah' logfile Thanks to this command for every line that has 'blah' in it, I get the output of the line that contains 'blah' and the next line that follows in the logfile. It might be a simple …

Read More