Celery: Worker with concurrency and reserved tasks only running 1 task

concurrency celery django-celery
steve-gregory picture steve-gregory · Jul 1, 2014 · Viewed 8k times · Source

Some of the tasks in my code were taking longer and longer to execute.

Upon inspection I noticed that although I have my worker node set to concurrency 6, and 6 processes exist to 'do work', but only 1 task is shown under 'running tasks'. Here is a little visual proof:

Here are the worker options:

The worker options, as seen by flower

And here is the task tab for that worker with only 1 running process:

enter image description here

I have found that if I restart celery, the concurrency is once again respected and i will see >1 running task, but after some amount of time/tasks it reverts back to this behavior..

Any ideas for fixing this intermittent problem?

Answer

Sébastien Deprez picture Sébastien Deprez · Jul 2, 2014

I'm not sure if it's your use case, but I ran into similar problems when I had a mix of long and short tasks. Basically what happened is that at some point a process could start a very long running task, while prefetching a few other tasks, preventing them from being consumed by other processes. So I disabled the prefetching stuff, which is useful only if you're running a lot of short tasks.

To disable the prefetch, you need Celery 3.1+ and the Ofair setting, for instance:

celery -A proj worker -l info -Ofair

Related questions

Celery difference between concurrency, workers and autoscaling

In my /etc/defaults/celeryd config file, I've set: CELERYD_NODES="agent1 agent2 agent3 agent4 agent5 agent6 agent7 agent8" CELERYD_OPTS="--autoscale=10,3 --concurrency=5" I understand that the daemon spawns 8 celery workers, but I'm fully not sure what autoscale and concurrency …

Read More
Celery Beat: Limit to single task instance at a time

I have celery beat and celery (four workers) to do some processing steps in bulk. One of those tasks is roughly along the lines of, "for each X that hasn't had a Y created, create a Y." The task is …

Read More
How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More