Downloading file with Python mechanize

python web-scraping mechanize
user1137778 picture user1137778 · Jun 12, 2012 · Viewed 13.5k times · Source

I am trying to download a file from a website using python and mechanize. My current code successfully logs on to the website and opens the page that contains the download link.

The download link is: https://www.lendingclub.com/browse/browseNotesRawDataV2.action

The info for the link is: 

Link(base_url='https://www.lendingclub.com/browse/browse.action', url='/browse/browseNotesRawDataV2.action', text='', tag='a', attrs=[('class', 'master_pngfix'), ('id', 'browseDownloadAllLink'), ('href', '/browse/browseNotesRawDataV2.action')])

I use the follow_link method to click on the link:

br = mechanize.Browser()
br.follow_link(url='/browse/browseNotesRawDataV2.action')

However, nothing happens and no file is downloaded. When I open the link in my browser when I'm logged on, it pauses for a few seconds and downloads the file.

How can I download the file using Python?

Answer

user1137778 picture user1137778 · Jun 13, 2012

For anyone who's interested, this was the solution:

br.retrieve('https://www.lendingclub.com/browse/browseNotesRawDataV2.action','l‌​oans.csv')[0]

Related questions

How to handle IncompleteRead: in python

I am trying to fetch some data from a website. However it returns me incomplete read. The data I am trying to get is a huge set of nested links. I did some research online and found that this might …

Read More
Scraping Data from Facebook with Python

I've been trying for several day now (unsuccessfully) to scrape cities from about 500 Facebook URLs. However, Facebook handles its data in a very strange way and I can't figure out what's going on under the hood to understand what I …

Read More
Using Mechanize (Python) to fill form

I want to fill the form on this page using python mechanize and then record the response. How should I do it? When I search for forms on this page using the following code, it shows the form only for …

Read More