Scrapy Shell - How to change USER_AGENT

python shell scrapy agent
dfriestedt picture dfriestedt · Aug 21, 2014 · Viewed 14k times · Source

I have a fully functioning scrapy script to extract data from a website. During setup, the target site banned me based on my USER_AGENT information. I subsequently added a RotateUserAgentMiddleware to rotate the USER_AGENT randomly. This works great.

However, now when I trying to use the scrapy shell to test xpath and css requests, I get a 403 error. I'm sure this is because the USER_AGENT of the scrapy shell is defaulting to some value the target site has blacklisted.

Question: is it possible to fetch a URL in the scrapy shell with a different USER_AGENT than the default?

fetch('http://www.test') [add something ?? to change USER_AGENT]

Thx

Answer

marven picture marven · Aug 22, 2014

scrapy shell -s USER_AGENT='custom user agent' 'http://www.example.com'

Related questions

How to call an external command?

How do you call an external command (as if I'd typed it at the Unix shell or Windows command prompt) from within a Python script?

Read More
Running shell command and capturing the output

I want to write a function that will execute a shell command and return its output as a string, no matter, is it an error or success message. I just want to get the same result that I would have …

Read More
Why do people write #!/usr/bin/env python on the first line of a Python script?

It seems to me like the files run the same without that line.

Read More