How can I defer the execution of Celery tasks?

python django celery django-celery
Mridang Agarwalla picture Mridang Agarwalla · Oct 22, 2012 · Viewed 22.4k times · Source

I have a small script that enqueues tasks for processing. This script makes a whole lot of database queries to get the items that should be enqueued. The issue I'm facing is that the celery workers begin picking up the tasks as soon as it is enqueued by the script. This is correct and it is the way celery is supposed to work but this often leads to deadlocks between my script and the celery workers.

Is there a way I could enqueue all my tasks from the script but delay execution until the script has completed or until a fixed time delay?

I couldn't find this in the documentation of celery or django-celery. Is this possible?

Currently as a quick-fix I've thought of adding all the items to be processed into a list and when my script is done executing all the queries, I can simply iterate over the list and enqueue the tasks. Maybe this would resolve the issue but when you have thousands of items to enqueue, this might be a bad idea.

Answer

mher picture mher · Oct 22, 2012

eta/countdown options enable to delay the task execution:

http://docs.celeryproject.org/en/master/userguide/calling.html#eta-and-countdown

Related questions

Django Celery Logging Best Practice

I'm trying to get Celery logging working with Django. I have logging set-up in settings.py to go to console (that works fine as I'm hosting on Heroku). At the top of each module, I have: import logging logger = logging.…

Read More
Starting Celery: AttributeError: 'module' object has no attribute 'celery'

I try to start a Celery worker server from a command line: celery -A tasks worker --loglevel=info The code in tasks.py: import os os.environ[ 'DJANGO_SETTINGS_MODULE' ] = "proj.settings" from celery import task @task() def add_photos_…

Read More
How can I run a celery periodic task from the shell manually?

I'm using celery and django-celery. I have defined a periodic task that I'd like to test. Is it possible to run the periodic task from the shell manually so that I view the console output?

Read More