How to remove non UTF-8 characters from text file

linux bash text utf-8 character-encoding
Hakim picture Hakim · Oct 21, 2012 · Viewed 103.5k times · Source

I have a bunch of Arabic, English, Russian files which are encoded in utf-8. Trying to process these files using a Perl script, I get this error:

Malformed UTF-8 character (fatal)

Manually checking the content of these files, I found some strange characters in them. Now I'm looking for a way to automatically remove these characters from the files.

Is there anyway to do it?

Answer

Palantir picture Palantir · Jun 11, 2013

This command:

iconv -f utf-8 -t utf-8 -c file.txt

will clean up your UTF-8 file, skipping all the invalid characters.

-f is the source format
-t the target format
-c skips any invalid sequence

Related questions

Quick unix command to display specific lines in the middle of a file?

Trying to debug an issue with a server and my only log file is a 20GB log file (with no timestamps even! Why do people use System.out.println() as logging? In production?!) Using grep, I've found an area of …

Read More
Looping through the content of a file in Bash

How do I iterate through each line of a text file with Bash? With this script: echo "Start!" for p in (peptides.txt) do echo "${p}" done I get this output on the screen: Start! ./runPep.sh: line 3: syntax error …

Read More
How to create a file in Linux from terminal window?

What's the easiest way to create a file in Linux terminal?

Read More