Using iconv to convert from UTF-16BE to UTF-8 without BOM

text utf-8 utf-16 iconv
Edward Samson picture Edward Samson · Jul 20, 2012 · Viewed 9.5k times · Source

I'm trying to convert a UTF-16BE encoded file (byte order mark: 0xFE 0xFF) to UTF-8 using iconv like so:

iconv -f UTF-16BE -t UTF-8 myfile.txt

The resulting output, however, has the UTF-8 byte order mark (0xEF 0xBB 0xBF) and that is not what I need. Is there a way to tell iconv (or is there an equivalent encoding) to not put a BOM in the UTF-8 result?

Answer

Keith Thompson picture Keith Thompson · Jul 20, 2012

Experiment shows that indicating UTF-16 rather than UTF-16BE does what you want:

iconv -f UTF-16 -t UTF-8 myfile.txt

Related questions

Best way to convert text files between character sets?

What is the fastest, easiest tool or method to convert text files between character sets? Specifically, I need to convert from UTF-8 to ISO-8859-15 and vice versa. Everything goes: one-liners in your favorite scripting language, command-line tools or other …

Read More
How to remove non UTF-8 characters from text file

I have a bunch of Arabic, English, Russian files which are encoded in utf-8. Trying to process these files using a Perl script, I get this error: Malformed UTF-8 character (fatal) Manually checking the content of these files, I found …

Read More
How can I find the character code of a special character in my text editor?

When pasting text from outside sources into a plain-text editor (e.g. TextMate or Sublime Text 2) a common problem is that special characters are often pasted in as well. Some of these characters render fine, but depending on the source, …

Read More