How do I access the scrapy settings in settings.py from the item pipeline. The documentation mentions it can be accessed through the crawler in extensions, but I don't see how to access the crawler in the pipelines.
UPDATE (2020-10-28)
Please note that this answer is now ~ 7years old, so it's validity can no longer be ensured.
The way to access your Scrapy settings (as defined in settings.py
) from within your_spider.py
is simple. All other answers are way too complicated. The reason for this is the very poor maintenance of the Scrapy documentation, combined with many recent updates & changes. Neither in the "Settings" documentation "How to access settings", nor in the "Settings API" have they bothered giving any workable example. Here's an example, how to get your current USER_AGENT string.
Just add the following lines to your_spider.py
:
# To get your settings from (settings.py):
from scrapy.utils.project import get_project_settings
...
class YourSpider(BaseSpider):
...
def parse(self, response):
...
settings = get_project_settings()
print "Your USER_AGENT is:\n%s" % (settings.get('USER_AGENT'))
...
As you can see, there's no need to use @classmethod
or re-define the from_crawler()
or __init__()
functions. Hope this helps.
PS. I'm still not sure why using from scrapy.settings import Settings
doesn't work the same way, since it would be the more obvious choice of import?