How to route a chain of tasks to a specific queue in celery?

python rabbitmq celery chain
mpaf picture mpaf · Feb 19, 2013 · Viewed 8.9k times · Source

When I route a task to a particular queue it works:

task.apply_async(queue='beetroot')

But if I create a chain:

chain = task | task

And then I write

chain.apply_async(queue='beetroot')

It seems to ignore the queue keyword and assigns to the default 'celery' queue.

It would be nice if celery supported routing in chains - all tasks executed sequentially in the same queue.

Answer

bigjools picture bigjools · Jun 10, 2014

I do it like this:

subtask = task.s(*myargs, **mykwargs).set(queue=myqueue)
mychain = celery.chain(subtask, subtask2, ...)
mychain.apply_async()

Related questions

Why use Celery instead of RabbitMQ?

From my understanding, Celery is a distributed task queue, which means the only thing that it should do is dispatching tasks/jobs to others servers and get the result back. RabbitMQ is a message queue, and nothing more. However, a …

Read More
Celery does not release memory

It looks like celery does not release memory after task finished. Every time a task finishes, there would be 5m-10m memory leak. So with thousands of tasks, soon it will use up all memory. BROKER_URL = 'amqp://user@localhost:5672/…

Read More
Celery: When should you choose Redis as a message broker over RabbitMQ?

My rough understanding is that Redis is better if you need the in-memory key-value store feature, however I am not sure how that has anything to do with distributing tasks? Does that mean we should use Redis as a message …

Read More