"for line in..." results in UnicodeDecodeError: 'utf-8' codec can't decode byte

python python-3.x character-encoding
SujitS picture SujitS · Oct 31, 2013 · Viewed 544.9k times · Source

Here is my code, 

for line in open('u.item'):
#read each line

whenever I run this code it gives the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 2892: invalid continuation byte

I tried to solve this and add an extra parameter in open(), the code looks like;

for line in open('u.item', encoding='utf-8'):
#read each line

But again it gives the same error. what should I do then! Please help.

Answer

SujitS picture SujitS · Oct 31, 2013

As suggested by Mark Ransom, I found the right encoding for that problem. The encoding was "ISO-8859-1", so replacing open("u.item", encoding="utf-8") with open('u.item', encoding = "ISO-8859-1") will solve the problem.

Related questions

Best way to convert string to bytes in Python 3?

There appear to be two different ways to convert a string to bytes, as seen in the answers to TypeError: 'str' does not support the buffer interface Which of these methods would be better or more Pythonic? Or is it …

Read More
What's "ANSI_X3.4-1968" encoding?

See following output on my system: [STEP 101] # python3 -c 'import sys; print(sys.stdout.encoding)' ANSI_X3.4-1968 [STEP 102] # [STEP 103] # locale LANG=C LANGUAGE=en_US:en LC_CTYPE="C" LC_NUMERIC="C" LC_TIME="C" LC_COLLATE="C" …

Read More
Convert bytes to a string

I'm using this code to get standard output from an external program: >>> from subprocess import * >>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0] The communicate() method returns an array of bytes: >>> …

Read More