How to parse malformed HTML in python

python html lxml
lorenzov picture lorenzov · May 24, 2009 · Viewed 7.4k times · Source

I need to browse the DOM tree of a parsed HTML document.

I'm using uTidyLib before parsing the string with lxml

a = tidy.parseString(html_code, options) dom = etree.fromstring(str(a))

sometimes I get an error, it seems that tidylib is not able to repair malformed html.

how can I parse every HTML file without getting an error (parsing only some parts of files that can not be repaired)?

Answer

dbr picture dbr · May 24, 2009

Beautiful Soup does a good job with invalid/broken HTML

>>> from BeautifulSoup import BeautifulSoup
>>> soup = BeautifulSoup("<htm@)($*><body><table <tr><td>hi</tr></td></body><html")
>>> print soup.prettify()
<htm>
 <body>
  <table>
   <tr>
    <td>
     hi
    </td>
   </tr>
  </table>
 </body>
</htm>

Related questions

How to Pretty Print HTML to a file, with indentation

I am using lxml.html to generate some HTML. I want to pretty print (with indentation) my final result into an html file. How do I do that? This is what I have tried and got till now (I am …

Read More
How to use lxml to find an element by text?

Assume we have the following html: <html> <body> <a href="/1234.html">TEXT A</a> <a href="/3243.html">TEXT B</a> <a href="/7445.html">TEXT C</…

Read More
Set lxml as default BeautifulSoup parser

I'm working on a web scraping project and have ran into problems with speed. To try to fix it, I want to use lxml instead of html.parser as BeautifulSoup's parser. I've been able to do this: soup = bs4.BeautifulSoup(…

Read More