How to bypass cloudflare bot/ddos protection in Scrapy?

javascript python cookies scrapy
Kulbi picture Kulbi · Oct 21, 2015 · Viewed 34.8k times · Source

I used to scrape e-commerce webpage occasionally to get product prices information. I have not used the scraper built using Scrapy in a while and yesterday was trying to use it - I run into a problem with bot protection.

It is using CloudFlare’s DDOS protection which is basically using JavaScript evaluation to filter out the browsers (and therefore scrapers) with JS disabled. Once the function is evaluated, the response with calculated number is generated. In return, service sends back two authentication cookies which attached to each request allow to normally crawl the site. Here's the description of how it works.

I have also found a cloudflare-scrape Python module that uses external JS evaluation engine to calculate the number and send the request back to server. I'm not sure how to integrate it into Scrapy though. Or maybe there's a smarter way without using JS execution? In the end, it's a form...

I'd apriciate any help.

Answer

Kulbi picture Kulbi · Oct 22, 2015

So I executed JavaScript using Python with help of cloudflare-scrape.

To your scraper, you need to add the following code:

def start_requests(self):
  for url in self.start_urls:
    token, agent = cfscrape.get_tokens(url, 'Your prefarable user agent, _optional_')
    yield Request(url=url, cookies=token, headers={'User-Agent': agent})

alongside parsing functions. And that's it!

Of course, you need to install cloudflare-scrape first and import it to your spider. You also need a JS execution engine installed. I had Node.JS already, no complaints.

Related questions

"The owner of this website has banned your access based on your browser's signature" ... on a url request in a python program

When doing a simple request, on python (Entought Canopy to be precise), with urllib2, the server denies me access : data = urllib.urlopen(an url i cannot post because of reputation, params) print data.read() Error: Access denied | play.pokemonshowdown.com …

Read More
Django Template Variables and Javascript

When I render a page using the Django template renderer, I can pass in a dictionary variable containing various values to manipulate them in the page using {{ myVar }}. Is there a way to access the same variable in Javascript (perhaps …

Read More
Call Python function from JavaScript code

I'd like to call a Python function from JavaScript code, because there isn't an alternative in JavaScript for doing what I want. Is this possible? Could you adjust the below snippet to work? JavaScript code: var tag = document.getElementsByTagName("p")[0]; …

Read More