Python asyncio, futures and yield from

Question

Consider the following program (running on CPython 3.4.0b1):

import math
import asyncio
from asyncio import coroutine

@coroutine
def fast_sqrt(x):
   future = asyncio.Future()
   if x >= 0:
      future.set_result(math.sqrt(x))
   else:
      future.set_exception(Exception("negative number"))
   return future


def slow_sqrt(x):
   yield from asyncio.sleep(1)
   future = asyncio.Future()
   if x >= 0:
      future.set_result(math.sqrt(x))
   else:
      future.set_exception(Exception("negative number"))
   return future


@coroutine
def run_test():
   for x in [2, -2]:
      for f in [fast_sqrt, slow_sqrt]:
         try:
            future = yield from f(x)
            print("\n{} {}".format(future, type(future)))
            res = future.result()
            print("{} result: {}".format(f, res))
         except Exception as e:
            print("{} exception: {}".format(f, e))


loop = asyncio.get_event_loop()
loop.run_until_complete(run_test())

I have 2 (related) questions:

1. Even with the decorator on fast_sqrt, Python seems to optimize away the Future created in fast_sqrt altogether, and a plain float is returned. Which then blows up in run_test() in the yield from

2. Why do I need to evaluate future.result() in run_test to retrieve the value of fire the exception? The docs say that yield from <future> "suspends the coroutine until the future is done, then returns the future’s result, or raises an exception". Why do I manually need to retieve the future's result?

Here is what I get:

oberstet@COREI7 ~/scm/tavendo/infrequent/scratchbox/python/asyncio (master)
$ python3 -V
Python 3.4.0b1
oberstet@COREI7 ~/scm/tavendo/infrequent/scratchbox/python/asyncio (master)
$ python3 test3.py

1.4142135623730951 <class 'float'>
<function fast_sqrt at 0x00B889C0> exception: 'float' object has no attribute 'result'

Future<result=1.4142135623730951> <class 'asyncio.futures.Future'>
<function slow_sqrt at 0x02AC8810> result: 1.4142135623730951
<function fast_sqrt at 0x00B889C0> exception: negative number

Future<exception=Exception('negative number',)> <class 'asyncio.futures.Future'>
<function slow_sqrt at 0x02AC8810> exception: negative number
oberstet@COREI7 ~/scm/tavendo/infrequent/scratchbox/python/asyncio (master)

Ok, I found the "issue". The yield from asyncio.sleep in slow_sqrt will make it a coroutine automatically. The waiting needs to be done differently:

def slow_sqrt(x):
   loop = asyncio.get_event_loop()
   future = asyncio.Future()
   def doit():
      if x >= 0:
         future.set_result(math.sqrt(x))
      else:
         future.set_exception(Exception("negative number"))
   loop.call_later(1, doit)
   return future

All 4 variants are here.

Answer 1

Regarding #1: Python does no such thing. Note that the fast_sqrt function you've written (i.e. before any decorators) is not a generator function, coroutine function, task, or whatever you want to call it. It's an ordinary function running synchronously and returning what you write after the return statement. Depending on the presence of @coroutine, very different things happen. It's just bad luck that both result the same error.

1. Without the decorator, fast_sqrt(x) runs like the ordinary function it is and returns a future of a float (regardless of context). That future is consumed by the future = yield from ..., leaving future a float (which doesn't have a result method).

2. With the decorator, the call f(x) goes through a wrapper function created by @coroutine. This wrapper function calls fast_sqrt and unpacks the resulting future for you, using the yield from <future> construction. Therefore, this wrapper function is itself a coroutine. Therefore, future = yield from ... waits on that coroutine and leaves future a float, again.

Regarding #2, yield from <future> does work (as explained above, you're using it when using the undecorated fast_sqrt), and you could also write:

future = yield from coro_returning_a_future(x)
res = yield from future

(Modulo that it doesn't work for fast_sqrt as written, and gains you no extra async-ness because the future is already done by the time it's returned from coro_returning_a_future.)

Your core problem seems to be that you confuse coroutines and futures. Both your sqrt implementations try to be async tasks resulting in futures. From my limited experience, that's not how one usually writes asyncio code. It allows you to pull both the construction of the future and the computation which the future stands for into two independent async tasks. But you don't do that (you return an already-finished future). And most of the time, this is not a useful concept: If you have to do some computation asynchronously, you either write it as a coroutine (which can be suspended) or you push it into another thread and communicate with it using yield from <future>. Not both.

To make the square root computation async, just write a regular coroutine doing the computation and return the result (the coroutine decorator will turn fast_sqrt into a task that runs asynchronously and can be waited on).

@coroutine
def fast_sqrt(x):
   if x >= 0:
      return math.sqrt(x)
   else:
      raise Exception("negative number")

@coroutine # for documentation, not strictly necessary
def slow_sqrt(x):
   yield from asyncio.sleep(1)
   if x >= 0:
      return math.sqrt(x)
   else:
      raise Exception("negative number")

...
res = yield from f(x)
assert isinstance(res, float)