Decoding HTML entities with Python

python unicode character-encoding content-type beautifulsoup
KeyboardInterrupt picture KeyboardInterrupt · Jul 30, 2009 · Viewed 21.5k times · Source

I'm trying to decode HTML entries from here NYTimes.com and I cannot figure out what I am doing wrong.

Take for example: 

"U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"

I've tried BeautifulSoup, decode('iso-8859-1'), and django.utils.encoding's smart_str without any success.

Answer

jfs picture jfs · Dec 21, 2013 
>>> from HTMLParser import HTMLParser
>>> print HTMLParser().unescape('U.S. Adviser’s Blunt Memo on Iraq: '
...                             'Time ‘to Go Home’')
U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’

The function is undocumented in Python 2. It is fixed in Python 3.4+: it is exposed as html.unescape() there.

Related questions

Convert Unicode to ASCII without errors in Python

My code just scrapes a web page, then converts it to Unicode. html = urllib.urlopen(link).read() html.encode("utf8","ignore") self.response.out.write(html) But I get a UnicodeDecodeError: Traceback (most recent call last): File "/Applications/GoogleAppEngineLauncher.app/…

Read More
Writing Unicode text to a text file?

I'm pulling data out of a Google doc, processing it, and writing it to a file (that eventually I will paste into a Wordpress page). It has some non-ASCII symbols. How can I convert these safely to symbols that can …

Read More
What is the difference between encode/decode?

I've never been sure that I understand the difference between str/unicode decode and encode. I know that str().decode() is for when you have a string of bytes that you know has a certain character encoding, given that encoding …

Read More