How to remove all special characters in Linux text

linux sed non-printable
vinllen picture vinllen · Mar 30, 2017 · Viewed 98.7k times · Source

vim pic How to remove the special characters shown as blue color in the picture 1 like: ^M, ^A, ^@, ^[. In my understanding, ^M is a windows newline character, I can use sed -i '/^M//g' to remove it, but it doesn't work to remove others. The command dos2unix doesn't work, neither. Are there exist any ways that I can use to remove them both?

Answer

heemayl picture heemayl · Mar 30, 2017

Remove everything except the printable characters (character class [:print:]), with sed:

sed $'s/[^[:print:]\t]//g' file.txt

[:print:] includes:

  • [:alnum:] (alpha-numerics)
  • [:punct:] (punctuations)
  • space

The ANSI C quoting ($'') is used for interpreting \t as literal tab inside $'' (in bash and alike).

Related questions

Delete empty lines using sed

I am trying to delete empty lines using sed: sed '/^$/d' but I have no luck with it. For example, I have these lines: xxxxxx yyyyyy zzzzzz and I want it to be like: xxxxxx yyyyyy zzzzzz What should …

Read More
Retrieve last 100 lines logs

I need to retrieve last 100 lines of logs from the log file. I tried the sed command sed -n -e '100,$p' logfilename Please let me know how can I change this command to specifically retrieve the last 100 lines.

Read More
How to remove all white spaces from a given text file

I want to remove all the white spaces from a given text file. Is there any shell command available for this ? Or, how to use sed for this purpose. I want something like below: $ cat hello.txt | sed .... I tried …

Read More