Python task queue alternatives and frameworks

python-3.x celery
Jens picture Jens · Oct 2, 2017 · Viewed 23.5k times · Source

There seem to be different implementations of task/job queues for Python 3:

  1. Celery, popular but apparently unmaintained and stale;
  2. RQ, of which I have little information;
  3. TaskTiger, similarly to RQ I know little about it;
  4. Huey , similarly to RQ I know little about it;
  5. WorQ had its last update in 2016.

Then there are “cloud” based solutions like Google’s Task Queue API or AWS’s Cloud Watch Events, but that’s more of a last resort.

For my project I am looking for a stable and active task queue implementation. I’ve used Celery for the past year, but the lack of support and non-attention to existing bugs is worrisome.

What alternatives exist?

Answer

Bogdan Popa picture Bogdan Popa · Nov 27, 2017

I wrote Dramatiq specifically because of my increasing frustration with Celery having used it professionally for years. Check out the motivation page the "why" and a feature comparison between Dramatiq, Celery and RQ. Some highlights:

  • actively developed and used in production
  • great docs
  • automatic retries
  • code auto-reload
  • locks and rate limiting
  • redis and rabbitmq support

Related questions

How to combine Celery with asyncio?

How can I create a wrapper that makes celery tasks look like asyncio.Task? Or is there a better way to integrate Celery with asyncio? @asksol, the creator of Celery, said this:: It's quite common to use Celery as a …

Read More
Cannot start Celery Worker (Kombu.asynchronous.timer)

I followed the first steps with Celery (Django) and trying to run a heavy process in the background. I have RabbitMQ server installed. However, when I try, celery -A my_app worker -l info it throws the following error File "&…

Read More
trouble in setting celery tasks backend in Python

I followed all the steps given in [ http://docs.celeryproject.org/en/latest/getting-started/first-steps-with-celery.html ] This is the code: from __future__ import absolute_import from celery import Celery #app = Celery('tasks', broker='pyamqp://guest@localhost//') app = Celery('tasks', …

Read More