Trying to delete non-ASCII characters only

regex linux sed tr
M_x_r picture M_x_r · Feb 23, 2013 · Viewed 19.2k times · Source

I am trying to manipulate a text file and remove non-ASCII characters from the text. I don't want to remove the line. I only want to remove the offending characters. I am trying to get the following expression to work:

sed '/[\x80-\xFF]/d'

Answer

speakr picture speakr · Feb 23, 2013

The suggested solutions may fail with specific version of sed, e.g. GNU sed 4.2.1.

Using tr:

tr -cd '[:print:]' < yourfile.txt

This will remove any characters not in [\x20-\x7e].

If you want to keep e.g. line feeds, just add \n:

tr -cd '[:print:]\n' < yourfile.txt

If you really want to keep all ASCII characters (even the control codes):

tr -cd '[:print:][:cntrl:]' < yourfile.txt

This will remove any characters not in [\x00-\x7f].

Related questions

How to use sed to replace the first space with an empty string

I am having trouble loading a space delimited text file into a table. The data in this text file is generated by teragen, and hence, is just dummy data, where there are only 2 columns, and the first column has values …

Read More
sed beginner: changing all occurrences in a folder

I need to do a regex find and replace on all the files in a folder (and its subfolders). What would be the linux shell command to do that? For example, I want to run this over all the files …

Read More
What's wrong with my lookahead regex in GNU sed?

This is what I'm doing (simplified example): gsed -i -E 's/^(?!foo)(.*)$/bar\1/' file.txt I'm trying to put bar in front of every line that doesn't start with foo. This is the error: gsed: -e expression #1, char 22: Invalid …

Read More