Scrapy get request url in parse

python-2.7 scrapy scrapyd
Goran picture Goran · Nov 19, 2013 · Viewed 43.3k times · Source

How can I get the request url in Scrapy's parse() function? I have a lot of urls in start_urls and some of them redirect my spider to homepage and as result I have an empty item. So I need something like item['start_url'] = request.url to store these urls. I'm using the BaseSpider.

Answer

Jagu picture Jagu · Jan 25, 2015

The 'response' variable that's passed to parse() has the info you want. You shouldn't need to override anything.

eg. (EDITED)

def parse(self, response):
    print "URL: " + response.request.url

Related questions

"OSError: [Errno 1] Operation not permitted" when installing Scrapy in OSX 10.11 (El Capitan) (System Integrity Protection)

I'm trying to install Scrapy Python framework in OSX 10.11 (El Capitan) via pip. The installation script downloads the required modules and at some point returns the following error: OSError: [Errno 1] Operation not permitted: '/tmp/pip-nIfswi-uninstall/System/Library/Frameworks/Python.…

Read More
Send Post Request in Scrapy

I am trying to crawl the latest reviews from google play store and to get that I need to make a post request. With the Postman, it works and I get desired response. but a post request in terminal gives …

Read More
python No module named service_identity

I tried to update scrapy and when I tried to check the version I got the following error C:\Windows\system32>scrapy version -v :0: UserWarning: You do not have a working installation of the service_identity module: 'No module …

Read More