Increase celery retry time each retry cycle

python celery celeryd
Gregor picture Gregor · Feb 27, 2012 · Viewed 7.5k times · Source

I do retries with celery like in the Docs-Example:

@task()
def add(x, y):
    try:
        ...
    except Exception, exc:
        add.retry(exc=exc, countdown=60)  # override the default and
                                          # retry in 1 minute

How can I increase the retry-countdown everytime the retry occurs for this job - e.g. 60 seconds, 2 minutes, 4 minutes and so on until the MaxRetriesExceeded is raised?

Answer

Lamp town guy picture Lamp town guy · Apr 11, 2014

Here is a simple way to create bigger delay each time the task is evaluated. This value is updated by celery itself so you don't need to manage anything yourself.

@task()
def add(x, y):
    try:
        ...
    except Exception as exc:
        raise add.retry(exc=exc, countdown=60 * add.request.retries)

Note: First task is repeated with countdown of 0. Because number of retries is 0 for the first run.

Related questions

How to start a Celery worker from a script/module __main__?

I've define a Celery app in a module, and now I want to start the worker from the same module in its __main__, i.e. by running the module with python -m instead of celery from the command line. I …

Read More
how to remove task from celery with redis broker?

I Have add some wrong task to a celery with redis broker but now I want to remove the incorrect task and I can't find any way to do this Is there some commands or some api to do this ?

Read More
Understanding celery task prefetching

I just found out about the configuration option CELERYD_PREFETCH_MULTIPLIER (docs). The default is 4, but (I believe) I want the prefetching off or as low as possible. I set it to 1 now, which is close enough to what I'm …

Read More