Converting ANSI to UTF-8 in shell

bash shell csv utf-8 iconv
Neringan picture Neringan · Nov 28, 2013 · Viewed 57.2k times · Source

I'm making a parser (1 csv to 3 csv) script and I have a problem. I am French so in my language I have letters like: é è à ....

A customer sent me a csv file that Linux recognizes as "unknown-8bit" (ansi I guess).

In my script, I'm writing 3 new csv files. But ViM creates them as ISO latin1 because it's close to what it got in the entry, but my é,è,à... are broken. I need UTF-8.

So I tried to convert the first ANSI csv to UTF-8 :

iconv -f "windows-1252" -t "UTF-8" import.csv -o import.csv

The problem is that it breaks my CSV. It's now on only one row. But my special chars are ok. Is there a way to convert ANSI to UTF-8 and keeping my rows?

Answer

Grzegorz Żur picture Grzegorz Żur · Nov 28, 2013

Put the output into another file. Don't overwrite the old one.

iconv -f "windows-1252" -t "UTF-8" import.csv -o new_import.csv

iconv fails when reading and writing to the same file.

Related questions

Turning multi-line string into single comma-separated

Let's say I have the following string: something1: +12.0 (some unnecessary trailing data (this must go)) something2: +15.5 (some more unnecessary trailing data) something4: +9.0 (some other unnecessary data) something1: +13.5 (blah blah blah) How do I turn that into simply +12.0,+15.5,+9.0,+13.5 in bash?

Read More
How to get the first column of every line from a CSV file?

How do get the first column of every line in an input CSV file and output to a new file? I am thinking using awk but not sure how.

Read More
Bash Script using mailx to mail attachment

I have a bash script that runs a query in postgres, it outputs to csv. I want to add to that script to use mailx to email that .csv file to a particular email. The problem I am having is …

Read More