Python and BeautifulSoup encoding issues

Question

I'm writing a crawler with Python using BeautifulSoup, and everything was going swimmingly till I ran into this site:

http://www.elnorte.ec/

I'm getting the contents with the requests library:

r = requests.get('http://www.elnorte.ec/')
content = r.content

If I do a print of the content variable at that point, all the spanish special characters seem to be working fine. However, once I try to feed the content variable to BeautifulSoup it all gets messed up:

soup = BeautifulSoup(content)
print(soup)
...
<a class="blogCalendarToday" href="/component/blog_calendar/?year=2011&amp;month=08&amp;day=27&amp;modid=203" title="1009 artículos en este día">
...

It's apparently garbling up all the spanish special characters (accents and whatnot). I've tried doing content.decode('utf-8'), content.decode('latin-1'), also tried messing around with the fromEncoding parameter to BeautifulSoup, setting it to fromEncoding='utf-8' and fromEncoding='latin-1', but still no dice.

Any pointers would be much appreciated.

Answer 1

In your case this page has wrong utf-8 data which confuses BeautifulSoup and makes it think that your page uses windows-1252, you can do this trick:

soup = BeautifulSoup.BeautifulSoup(content.decode('utf-8','ignore'))

by doing this you will discard any wrong symbols from the page source and BeautifulSoup will guess the encoding correctly.

You can replace 'ignore' by 'replace' and check text for '?' symbols to see what has been discarded.

Actually it's a very hard task to write crawler which can guess page encoding every time with 100% chance(Browsers are very good at this nowadays), you can use modules like 'chardet' but, for example, in your case it will guess encoding as ISO-8859-2, which is not correct too.

If you really need to be able to get encoding for any page user can possibly supply - you should either build a multi-level(try utf-8, try latin1, try etc...) detection function(like we did in our project) or use some detection code from firefox or chromium as C module.