How to convert a string from CP-1251 to UTF-8?

python utf-8 wxpython cp1251
jsnjack picture jsnjack · Sep 26, 2011 · Viewed 52k times · Source

I'm using mutagen to convert ID3 tags data from CP-1251/CP-1252 to UTF-8. In Linux there is no problem. But on Windows, calling SetValue() on a wx.TextCtrl produces the error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 0: ordinal not in range(128)

The original string (assumed to be CP-1251 encoded) that I'm pulling from mutagen is:

u'\xc1\xe5\xeb\xe0\xff \xff\xe1\xeb\xfb\xed\xff \xe3\xf0\xee\xec\xf3'

I've tried converting this to UTF-8:

dd = d.decode('utf-8')

...and even changing the default encoding from ASCII to UTF-8:

sys.setdefaultencoding('utf-8')

...But I get the same error.

Answer

Johannes Charra picture Johannes Charra · Sep 26, 2011

If you know for sure that you have cp1251 in your input, you can do

d.decode('cp1251').encode('utf8')

Related questions

Working with UTF-8 encoding in Python source

Consider: $ cat bla.py u = unicode('d…') s = u.encode('utf-8') print s $ python bla.py File "bla.py", line 1 SyntaxError: Non-ASCII character '\xe2' in file bla.py on line 1, but no encoding declared; see http://…

Read More
Unicode (UTF-8) reading and writing to files in Python

I'm having some brain failure in understanding reading and writing text to a file (Python 2.4). # The string, which has an a-acute in it. ss = u'Capit\xe1n' ss8 = ss.encode('utf8') repr(ss), repr(ss8) ("u'Capit\xe1n'", "'Capit\…

Read More
error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

https://github.com/affinelayer/pix2pix-tensorflow/tree/master/tools An error occurred when compiling "process.py" on the above site. python tools/process.py --input_dir data -- operation resize --outp ut_dir data2/resize data/0.jpg -> data2/…

Read More