How to disable robots.txt when you launch scrapy shell?

python scrapy web-crawler robots.txt scrapy-shell
DARDAR SAAD picture DARDAR SAAD · Nov 26, 2016 · Viewed 8.2k times · Source

I use Scrapy shell without problems with several websites, but I find problems when the robots (robots.txt) does not allow access to a site. How can I disable robots detection by Scrapy (ignored the existence)? Thank you in advance. I'm not talking about the project created by Scrapy, but Scrapy shell command: scrapy shell 'www.example.com'

Answer

Granitosaurus picture Granitosaurus · Nov 27, 2016

If you run scrapy from project directory scrapy shell will use the projects settings.py. If you run outside of the project scrapy will use default settings. However you can override and add settings via --set flag.
So to turn off ROBOTSTXT_OBEY setting you can simply: 

scrapy shell http://stackoverflow.com --set="ROBOTSTXT_OBEY=False"

Related questions

Difference between BeautifulSoup and Scrapy crawler?

I want to make a website that shows the comparison between amazon and e-bay product price. Which of these will work better and why? I am somewhat familiar with BeautifulSoup but not so much with Scrapy crawler.

Read More
How to run Scrapy from within a Python script

I'm new to Scrapy and I'm looking for a way to run it from a Python script. I found 2 sources that explain this: http://tryolabs.com/Blog/2011/09/27/calling-scrapy-python-script/ http://snipplr.com/view/67006/using-scrapy-from-a-script/ I can't figure out where I should …

Read More
How to pass a user defined argument in scrapy spider

I am trying to pass a user defined argument to a scrapy's spider. Can anyone suggest on how to do that? I read about a parameter -a somewhere but have no idea how to use it.

Read More