Unicode Encode Error when writing pandas df to csv

python pandas export-to-csv python-unicode
collarblind picture collarblind · Jul 10, 2015 · Viewed 46.2k times · Source

I cleaned 400 excel files and read them into python using pandas and appended all the raw data into one big df.

Then when I try to export it to a csv:

df.to_csv("path",header=True,index=False)

I get this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xc7' in position 20: ordinal not in range(128)

Can someone suggest a way to fix this and what it means?

Thanks

Answer

unutbu picture unutbu · Jul 10, 2015

You have unicode values in your DataFrame. Files store bytes, which means all unicode have to be encoded into bytes before they can be stored in a file. You have to specify an encoding, such as utf-8. For example, 

df.to_csv('path', header=True, index=False, encoding='utf-8')

If you don't specify an encoding, then the encoding used by df.to_csv defaults to ascii in Python2, or utf-8 in Python3.

Related questions

How to specify date format when using pandas.to_csv?

The default output format of to_csv() is: 12/14/2012 12:00:00 AM I cannot figure out how to output only the date part with specific format: 20121214 or date and time in two separate columns in the csv file: 20121214, 084530 The documentation is too brief …

Read More
Pandas to_csv() checking for overwrite

When I am analyzing data, I save my dataframes into a csv-file and use pd.to_csv() for that. However, the function (over)writes the new file, without checking whether there exists one with the same name. Is there a …

Read More
Apply GZIP compression to a CSV in Python Pandas

I am trying to write a dataframe to a gzipped csv in python pandas, using the following: import pandas as pd import datetime import csv import gzip # Get data (with previous connection and script variables) df = pd.read_sql_query(…

Read More