UnicodeDecodeError, invalid continuation byte

python unicode decode
RuiDC picture RuiDC · Apr 5, 2011 · Viewed 739.6k times · Source

Why is the below item failing? Why does it succeed with "latin-1" codec?

o = "a test of \xe9 char" #I want this to remain a string as this is what I am receiving
v = o.decode("utf-8")

Which results in:

 Traceback (most recent call last):  
 File "<stdin>", line 1, in <module>  
 File "C:\Python27\lib\encodings\utf_8.py",
 line 16, in decode
     return codecs.utf_8_decode(input, errors, True) UnicodeDecodeError:
 'utf8' codec can't decode byte 0xe9 in position 10: invalid continuation byte

Answer

Mazen Aly picture Mazen Aly · Jul 18, 2015

I had the same error when I tried to open a CSV file by pandas.read_csv method.

The solution was change the encoding to latin-1:

pd.read_csv('ml-100k/u.item', sep='|', names=m_cols , encoding='latin-1')

Related questions

How to fix ''UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 29815: character maps to <undefined>''?

At the moment, I am trying to get a Python 3 program to do some manipulations with a text file filled with information, through the Spyder IDE/GUI. However, when trying to read the file I get the following error: File "&…

Read More
I don't understand encode and decode in Python (2.7.3)

I tried to understand by myself encode and decode in Python but nothing is really clear for me. str.encode([encoding,[errors]]) str.decode([encoding,[errors]]) First, I don't understand the need of the "encoding" parameter in these two functions. …

Read More
'ascii' codec can't encode character at position * ord not in range(128)

There are a few threads on stackoverflow, but i couldn't find a valid solution to the problem as a whole. I have collected huge sums of textual data from the urllib read function and stored the same in pickle files. …

Read More