How to make mechanize wait for web-page 'full' load?

python mechanize
planetes853 picture planetes853 · Jan 12, 2012 · Viewed 11.9k times · Source

I want to scrape some web page which loads its components dynamically. This page has an onload script, and I can see the complete page 3-5 seconds after typing the URL into my browser.

The problem is, when I call br.open('URL'), the response is the web page at 0 seconds. There is a difference 3-5 seconds later between the HTML (which I want) and result of br.open('URL').

Answer

jcollado picture jcollado · Jan 12, 2012

The problem you're having is that the web page is rendered in your web browser through the javascript engine. However, mechanize doesn't have the ability to execute javascript on its own so, no matter how long you wait, you aren't going to get the HTML you're missing using just mechanize.

For more information about how scrape dynamically generated content, please have a look at this question.

Related questions

Unable to import a module that is definitely installed

After installing mechanize, I don't seem to be able to import it. I have tried installing from pip, easy_install, and via python setup.py install from this repo: https://github.com/abielr/mechanize. All of this to no avail, …

Read More
adding directory to sys.path /PYTHONPATH

I am trying to import a module from a particular directory. The problem is that if I use sys.path.append(mod_directory) to append the path and then open the python interpreter, the directory mod_directory gets added to …

Read More
How to avoid HTTP error 429 (Too Many Requests) python

I am trying to use Python to login to a website and gather information from several webpages and I get the following error: Traceback (most recent call last): File "extract_test.py", line 43, in <module> response=br.open(…

Read More