Using python Requests with javascript pages

Question

I am trying to use the Requests framework with python (http://docs.python-requests.org/en/latest/) but the page I am trying to get to uses javascript to fetch the info that I want.

I have tried to search on the web for a solution but the fact that I am searching with the keyword javascript most of the stuff I am getting is how to scrape with the javascript language.

Is there anyway to use the requests framework with pages that use javascript?

Answer 1

Good news: there is now a requests module that supports javascript: https://pypi.org/project/requests-html/

from requests_html import HTMLSession

session = HTMLSession()

r = session.get('http://www.yourjspage.com')

r.html.render()  # this call executes the js in the page

As a bonus this wraps BeautifulSoup, I think, so you can do things like

r.html.find('#myElementID').text

which returns the content of the HTML element as you'd expect.