Screen scraping: getting around "HTTP Error 403: request disallowed by robots.txt"

python screen-scraping beautifulsoup mechanize http-status-code-403
Diego picture Diego · May 17, 2010 · Viewed 41.5k times · Source

Is there a way to get around the following?

httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt

Is the only way around this to contact the site-owner (barnesandnoble.com).. i'm building a site that would bring them more sales, not sure why they would deny access at a certain depth.

I'm using mechanize and BeautifulSoup on Python2.6.

hoping for a work-around

Answer

Yuda Prawira picture Yuda Prawira · Oct 3, 2010

oh you need to ignore the robots.txt

br = mechanize.Browser()
br.set_handle_robots(False)

Related questions

BeautifulSoup: How do I extract all the <li>s from a list of <ul>s that contains some nested <ul>s?

My source code looks like: <h3>Header3 (Start here)</h3> <ul> <li>List items</li> <li>Etc...</li> </ul> <h3>Header 3</…

Read More
Best way for a beginner to learn screen scraping by Python

This might be one of those questions that are difficult to answer, but here goes: I don't consider my self programmer - but I would like to :-) I've learned R, because I was sick and tired of spss, and …

Read More
Beautiful Soup cannot find a CSS class if the object has other classes, too

if a page has <div class="class1"> and <p class="class1">, then soup.findAll(True, 'class1') will find them both. If it has <p class="class1 class2">, though, it will not be found. …

Read More