Beautiful Soup cannot find a CSS class if the object has other classes, too

python screen-scraping beautifulsoup
endolith picture endolith · Aug 7, 2009 · Viewed 15.1k times · Source

if a page has <div class="class1"> and <p class="class1">, then soup.findAll(True, 'class1') will find them both.

If it has <p class="class1 class2">, though, it will not be found. How do I find all objects with a certain class, regardless of whether they have other classes, too?

Answer

endolith picture endolith · Aug 7, 2009

Unfortunately, BeautifulSoup treats this as a class with a space in it 'class1 class2' rather than two classes ['class1','class2']. A workaround is to use a regular expression to search for the class instead of a string.

This works: 

soup.findAll(True, {'class': re.compile(r'\bclass1\b')})

Related questions

BeautifulSoup: How do I extract all the <li>s from a list of <ul>s that contains some nested <ul>s?

My source code looks like: <h3>Header3 (Start here)</h3> <ul> <li>List items</li> <li>Etc...</li> </ul> <h3>Header 3</…

Read More
Screen scraping: getting around "HTTP Error 403: request disallowed by robots.txt"

Is there a way to get around the following? httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt Is the only way around this to contact the site-owner (barnesandnoble.com).. i'm building a site that would bring them more …

Read More
Best way for a beginner to learn screen scraping by Python

This might be one of those questions that are difficult to answer, but here goes: I don't consider my self programmer - but I would like to :-) I've learned R, because I was sick and tired of spss, and …

Read More