How can I make scrapy crawl break and exit when encountering the first exception?

python exception scrapy
Udi picture Udi · Mar 1, 2012 · Viewed 10k times · Source

For development purposes, I would like to stop all scrapy crawling activity as soon a first exception (in a spider or a pipeline) occurs.

Any advice?

Answer

tokarev picture tokarev · Mar 8, 2016

Since 0.11, there is CLOSESPIDER_ERRORCOUNT:

An integer which specifies the maximum number of errors to receive before closing the spider. If the spider generates more than that number of errors, it will be closed with the reason closespider_errorcount. If zero (or non set), spiders won’t be closed by number of errors.

If it is set to 1, the spider will be closed on the first exception.

Related questions

How can I safely create a nested directory?

What is the most elegant way to check if the directory a file is going to be written to exists, and if not, create the directory using Python? Here is what I tried: import os file_path = "/my/directory/filename.…

Read More
Manually raising (throwing) an exception in Python

How can I raise an exception in Python so that it can later be caught via an except block?

Read More
How do I check if a variable exists?

I want to check if a variable exists. Now I'm doing something like this: try: myVar except NameError: # Do something. Are there other ways without exceptions?

Read More