Celery: per task concurrency limits (# of workers per task)?

celery django-celery
Parand picture Parand · Feb 6, 2012 · Viewed 10.6k times · Source

Is it possible to set the concurrency (the number of simultaneous workers) on a per-task level in Celery? I'm looking for something more fine-grained that CELERYD_CONCURRENCY (that sets the concurrency for the whole daemon).

The usage scenario is: I have a single celerlyd running different types of tasks with very different performance characteristics - some are fast, some very slow. For some I'd like to do as many as I can as quickly as I can, for others I'd like to ensure only one instance is running at any time (ie. concurrency of 1).

Answer

mher picture mher · Feb 7, 2012

You can use automatic routing to route tasks to different queues which will be processed by celery workers with different concurrency levels.

celeryd-multi start fast slow -c:slow 3 -c:fast 5

This command launches 2 celery workers listening fast and slow queues with 3 and 5 concurrency levels respectively.

CELERY_ROUTES = {"tasks.a": {"queue": "slow"}, "tasks.b": {"queue": "fast"}}

The tasks with type tasks.a will be processed by slow queue and tasks.b tasks by fast queue respectively.

Related questions

How to check task status in Celery?

How does one check whether a task is running in celery (specifically, I'm using celery-django)? I've read the documentation, and I've googled, but I can't see a call like: my_example_task.state() == RUNNING My use-case is that I have …

Read More
Django Celery Logging Best Practice

I'm trying to get Celery logging working with Django. I have logging set-up in settings.py to go to console (that works fine as I'm hosting on Heroku). At the top of each module, I have: import logging logger = logging.…

Read More
Starting Celery: AttributeError: 'module' object has no attribute 'celery'

I try to start a Celery worker server from a command line: celery -A tasks worker --loglevel=info The code in tasks.py: import os os.environ[ 'DJANGO_SETTINGS_MODULE' ] = "proj.settings" from celery import task @task() def add_photos_…

Read More