How do I grep for all non-ASCII characters?

regex unix unicode grep
pconrey picture pconrey · Jun 8, 2010 · Viewed 282.5k times · Source

I have several very large XML files and I'm trying to find the lines that contain non-ASCII characters. I've tried the following:

grep -e "[\x{00FF}-\x{FFFF}]" file.xml

But this returns every line in the file, regardless of whether the line contains a character in the range specified.

Do I have the syntax wrong or am I doing something else wrong? I've also tried:

egrep "[\x{00FF}-\x{FFFF}]" file.xml

(with both single and double quotes surrounding the pattern).

Answer

jerrymouse picture jerrymouse · Feb 22, 2012

You can use the command: 

grep --color='auto' -P -n "[\x80-\xFF]" file.xml

This will give you the line number, and will highlight non-ascii chars in red.

In some systems, depending on your settings, the above will not work, so you can grep by the inverse

grep --color='auto' -P -n "[^\x00-\x7F]" file.xml

Note also, that the important bit is the -P flag which equates to --perl-regexp: so it will interpret your pattern as a Perl regular expression. It also says that

this is highly experimental and grep -P may warn of unimplemented features.

Related questions

grep --ignore-case --only

grep fails when using both --ignore-case and --only-match options. Example: $ echo "abc" | grep -io abc abc $ echo "ABC" | grep -io abc $ But $ echo "abc" | grep -i abc abc $ echo "ABC" | grep -i abc ABC According to man page: -o, --only-matching …

Read More
linux find regex

I'm having trouble using the regex of the find command. Probably something I don't understand about escaping on the command line. Why are these not the same? find -regex '.*[1234567890]' find -regex '.*[[:digit:]]' Bash, Ubuntu

Read More
how to use sed, awk, or gawk to print only what is matched?

I see lots of examples and man pages on how to do things like search-and-replace using sed, awk, or gawk. But in my case, I have a regular expression that I want to run against a text file to extract …

Read More