How to combine Celery with asyncio?

python python-3.x asynchronous celery python-asyncio
max picture max · Oct 2, 2016 · Viewed 18.4k times · Source

How can I create a wrapper that makes celery tasks look like asyncio.Task? Or is there a better way to integrate Celery with asyncio?

@asksol, the creator of Celery, said this::

It's quite common to use Celery as a distributed layer on top of async I/O frameworks (top tip: routing CPU-bound tasks to a prefork worker means they will not block your event loop).

But I could not find any code examples specifically for asyncio framework.

Answer

John Moutafis picture John Moutafis · Apr 10, 2017

That will be possible from Celery version 5.0 as stated on the official site:

http://docs.celeryproject.org/en/4.0/whatsnew-4.0.html#preface

  1. The next major version of Celery will support Python 3.5 only, where we are planning to take advantage of the new asyncio library.
  2. Dropping support for Python 2 will enable us to remove massive amounts of compatibility code, and going with Python 3.5 allows us to take advantage of typing, async/await, asyncio, and similar concepts there’s no alternative for in older versions.

The above was quoted from the previous link.

So the best thing to do is wait for version 5.0 to be distributed!

In the meantime, happy coding :)

Related questions

how to add a coroutine to a running asyncio loop?

How can one add a new coroutine to a running asyncio loop? Ie. one that is already executing a set of coroutines. I guess as a workaround one could wait for existing coroutines to complete and then initialize a new …

Read More
How to create an event loop with rolling coroutines running on it forever?

In order to prevent from context switching, I want to create a big loop to serve both the network connections and some routines. Here's the implementation for normal functions: import asyncio import time def hello_world(loop): print('Hello World') …

Read More
Simplest async/await example possible in Python

I've read many examples, blog posts, questions/answers about asyncio / async / await in Python 3.5+, many were complex, the simplest I found was probably this one. Still it uses ensure_future, and for learning purposes about asynchronous programming in Python, I …

Read More