How to give delay between each requests in scrapy?

Question

I don't want to crawl simultaneously and get blocked. I would like to send one request per second.

Answer 1

There is a setting for that:

DOWNLOAD_DELAY

Default: 0

The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard.

DOWNLOAD_DELAY = 0.25    # 250 ms of delay

Read the docs: https://doc.scrapy.org/en/latest/index.html