iconv: Converting from Windows ANSI to UTF-8 with BOM

unicode iconv
user531912 picture user531912 · Dec 6, 2010 · Viewed 53.3k times · Source

I want to use iconv to convert files on my Mac. The goal is to go from "Windows ANSI" to "whatever Windows Notepad saves, if you tell it to use UFT8".

This is what I want:

$ file names.csv 
names.csv: UTF-8 Unicode (with BOM) text, with CRLF line terminators

This is what I use:

$ iconv -f CP1252 -t UTF-8  names.csv > names.utf8.csv

This is what I get (not what I want):

$ file names.utf8.csv 
names.utf8.csv: UTF-8 Unicode text, with CRLF line terminators

How do I get the BOM?

Answer

borrible picture borrible · Dec 6, 2010

You can add it manually by first echoing the bytes into the file:

echo -ne '\xEF\xBB\xBF' > names.utf8.csv

and then concatenating your required information at the end:

iconv -f CP1252 -t UTF-8  names.csv >> names.utf8.csv

Note the >> rather than >.

Related questions

Converting a \u escaped Unicode string to ASCII

After reading all about iconv and Encoding, I am still confused. I am scraping the source of a web page I have a string that looks like this: 'pretty\u003D\u003Ebig' (displayed in the R console as 'pretty\\\…

Read More
How to use iconv(3) to convert wide string to UTF-8?

I'm trying to use iconv(3) to convert a wide-character string to UTF-8 using the code below. When I run the below, the iconv call returns E2BIG, as if there were not enough bytes of space available in the output …

Read More
Converting Unicode characters into the equivalent ASCII ones

I need to "flatten out" a number of Unicode strings for the purposes of indexing and searching. For example, I need to convert GötheФ€ into ASCII. The last two characters have no close representations in ASCII so it's Ok …

Read More