is Scrapy single-threaded or multi-threaded?

python multithreading scrapy web-crawler
Gill Bates picture Gill Bates · Jul 15, 2014 · Viewed 13.7k times · Source

There are few concurrency settings in Scrapy, like CONCURRENT_REQUESTS. Does it mean, that Scrapy crawler is multi-threaded? So if I run scrapy crawl my_crawler it will literally fire multiple simultaneous requests in parallel? Im asking because, I've read that Scrapy is single-threaded.

Answer

famousgarkin picture famousgarkin · Jul 15, 2014

Scrapy is single-threaded, except the interactive shell and some tests, see source.

It's built on top of Twisted, which is single-threaded too, and makes use of it's own asynchronous concurrency capabilities, such as twisted.internet.interfaces.IReactorThreads.callFromThread, see source.

Related questions

How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More
Is there any way to kill a Thread?

Is it possible to terminate a running thread without setting/checking any flags/semaphores/etc.?

Read More
Run certain code every n seconds

Is there a way to, for example, print Hello World! every n seconds? For example, the program would go through whatever code I had, then once it had been 5 seconds (with time.sleep()) it would execute that code. I would …

Read More