Trying to remove non-printable characters (junk values) from a UNIX file

bash unix awk sed non-printing-characters
Pranav picture Pranav · Dec 22, 2015 · Viewed 23.1k times · Source

I am trying to remove non-printable character (for e.g. ^@) from records in my file. Since the volume to records is too big in the file using cat is not an option as the loop is taking too much time. I tried using 

sed -i 's/[^@a-zA-Z 0-9`~!@#$%^&*()_+\[\]\\{}|;'\'':",.\/<>?]//g' FILENAME

but still the ^@ characters are not removed. Also I tried using 

awk '{ sub("[^a-zA-Z0-9\"!@#$%^&*|_\[](){}", ""); print } FILENAME > NEW FILE

but it also did not help.

Can anybody suggest some alternative way to remove non-printable characters?

Used tr -cd but it is removing accented characters. But they are required in the file.

Answer

Tom Fenech picture Tom Fenech · Dec 22, 2015

Perhaps you could go with the complement of [:print:], which contains all printable characters:

tr -cd '[:print:]' < file > newfile

If your version of tr doesn't support multi-byte characters (it seems that many don't), this works for me with GNU sed (with UTF-8 locale settings):

sed 's/[^[:print:]]//g' file

Related questions

Bash tool to get nth line from a file

Is there a "canonical" way of doing that? I've been using head -n | tail -1 which does the trick, but I've been wondering if there's a Bash tool that specifically extracts a line (or a range of lines) from a …

Read More
Extract The Second Word In A Line From A Text File

I am currently in need to extract the second word (CRISTOBAL) in a line in a text file. * CRISTOBAL AL042014 08/05/14 12 UTC * The second word in the line "CRISTOBAL" will vary from day to day so, I just need to find …

Read More
cut or awk command to print first field of first row

I am trying print the first field of the first row of an output. Here is the case. I just need to print only SUSE from this output. # cat /etc/*release SUSE Linux Enterprise Server 11 (x86_64) VERSION = 11 PATCHLEVEL = 2 Tried with …

Read More