Scrapy or Selenium or Mechanize to scrape web data?

selenium-webdriver web-scraping scrapy mechanize
asit_dhal picture asit_dhal · Jan 5, 2014 · Viewed 7.3k times · Source

I want to scrape some data from a website.

Basically, the website has some tabular display and shows around 50 records. For more records, the user has to click some button which makes an ajax call get & show the next 50 records.

I have previous knowledge of Selenium webdriver(Python). I can do this very quickly in Selenium. But, Selenium is more kind of automation testing tool and it is very slow.

I did some R&D and found that using Scrapy or Mechanize, I can also do the same thing.

Should I go for Scrapy or Mechanize or Selenium for this ?

Answer

Severin picture Severin · Jan 21, 2014

I would recommend you to go with a combination of Mechanize and ExecJS (https://github.com/sstephenson/execjs) to execute any javascript requests you might come across. I have used those two gems in combination for quite some time now and they do a great job.

You should choose this instead of Selenium, because it it will be a lot faster compared to having to render the entire page in a headless browser.

Related questions

selenium with scrapy for dynamic page

I'm trying to scrape product information from a webpage, using scrapy. My to-be-scraped webpage looks like this: starts with a product_list page with 10 products a click on "next" button loads the next 10 products (url doesn't change between the two …

Read More
How can I download a file on a click event using selenium?

I am working on python and selenium. I want to download file from clicking event using selenium. I wrote following code. from selenium import webdriver from selenium.common.exceptions import NoSuchElementException from selenium.webdriver.common.keys import Keys browser = webdriver.…

Read More
Clicking on a link via selenium

I am trying to do some webscraping via Selenium. My question is very simple: How do you find a link and then how do you click on it? For instance: The following is the HTML that I am trying to …

Read More