Using boto to invoke lambda functions how do I do so asynchronously?

Question

SO I'm using boto to invoke my lambda functions and test my backend. I want to invoke them asynchronously. I have noted that "invoke_async" is deprecated and should not be used. Instead you should use "invoke" with an InvocationType of "Event" to do the function asynchronously.

I can't seem to figure out how to get the responses from the functions when they return though. I have tried the following:

payload3=b"""{
"latitude": 39.5732160891,
"longitude": -119.672918997,
"radius": 100
}"""

client = boto3.client('lambda')
for x in range (0, 5):
    response = client.invoke(
        FunctionName="loadSpotsAroundPoint",
        InvocationType='Event',
        Payload=payload3
    )
    time.sleep(15)
    print(json.loads(response['Payload'].read()))
    print("\n")

Even though I tell the code to sleep for 15 seconds, the response variable is still empty when I try and print it. If I change the invokation InvokationType to "RequestResponse" it all works fine and response variable prints, but this is synchronous. Am I missing something easy? How do i execute some code, for example print out the result, when the async invokation returns??

Thanks.

Answer 1

There is a difference between an 'async AWS lambda invocation' and 'async python code'. When you set the InvocationType to 'Event', by definition, it does not ever send back a response.

In your example, invoke() immediately returns None, and does not implicitly start up anything in the background to change that value at a later time (thank goodness!). So, when you look at the value of response 15 seconds later, it's still None.

It seems what you really want is the RequestResponse invocation type, with asynchronous Python code. You have a bunch of options to choose from, but my favorite is concurrent.futures. Another is threading.

Here's an example using concurrent.futures:

(If you're using Python2 you'll need to pip install futures)

from concurrent.futures import ThreadPoolExecutor
import json

payload = {...}

with ThreadPoolExecutor(max_workers=5) as executor:
    futs = []
    for x in xrange(0, 5):
        futs.append(
            executor.submit(client.invoke,
                FunctionName   = "loadSpotsAroundPoint",
                InvocationType = "RequestResponse",
                Payload        = bytes(json.dumps(payload))
            )
        )
    results = [ fut.result() for fut in futs ]

print results

Another pattern you might want to look into is to use the Event invocation type, and have your Lambda function push messages to SNS, which are then consumed by another Lambda function. You can check out a tutorial for SNS-triggered lambda functions here.