Should I use two asyncio event loops in one program?

Question

I want use the Python 3 asyncio module to create a server application. I use a main event loop to listen to the network, and when new data is received it will do some compute and send the result to the client. Does 'do some compute' need a new event loop? or can it use the main event loop?

Answer 1

You can do the compute work in the main event loop, but the whole event loop will be blocked while that happens - no other requests can be served, and anything else you have running in the event loop will be blocked. If this isn't acceptable, you probably want to run the compute work in a separate process, using BaseEventLoop.run_in_executor. Here's a very simple example demonstrating it:

import time
import asyncio
from concurrent.futures import ProcessPoolExecutor

def cpu_bound_worker(x, y):
    print("in worker")
    time.sleep(3)
    return x +y

@asyncio.coroutine
def some_coroutine():
    yield from asyncio.sleep(1)
    print("done with coro")

@asyncio.coroutine
def main():
    loop = asyncio.get_event_loop()
    loop.set_default_executor(ProcessPoolExecutor())
    asyncio.async(some_coroutine())
    out = yield from loop.run_in_executor(None, cpu_bound_worker, 3, 4)
    print("got {}".format(out))

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Output:

in worker
done with coro
got 7

cpu_bound_worker gets executed in a child process, and the event loop will wait for the result like it would any other non-blocking I/O operation, so it doesn't block other coroutines from running.