Set headers for scrapy shell request

scrapy scrapy-shell
octohedron picture octohedron · May 3, 2016 · Viewed 10.3k times · Source

I know that you can scrapy shell -s USER_AGENT='custom user agent' 'http://www.example.com' to change the USER_AGENT, but how do you add request headers?

Answer

eLRuLL picture eLRuLL · May 3, 2016

there is no current way to add headers directly on cli, but you could do something like:

$ scrapy shell
...
...
>>> from scrapy import Request
>>> req = Request('yoururl.com', headers={"header1":"value1"})
>>> fetch(req)

This will update the current shell information with that new request.

Related questions

scrapy xpath select elements by classname

I have followed How can I find an element by CSS class with XPath? which gives the selector to use for selecting elements by classname. The problem is when I use it it retrieves an empty result "[]" and I know …

Read More
Scrapy Shell and Scrapy Splash

We've been using scrapy-splash middleware to pass the scraped HTML source through the Splash javascript engine running inside a docker container. If we want to use Splash in the spider, we configure several required project settings and yield a Request …

Read More
How to disable robots.txt when you launch scrapy shell?

I use Scrapy shell without problems with several websites, but I find problems when the robots (robots.txt) does not allow access to a site. How can I disable robots detection by Scrapy (ignored the existence)? Thank you in advance. …

Read More