Get return value for multi-processing functions in python

TheStupidOne picture TheStupidOne · Jun 17, 2016 · Viewed 7.4k times · Source

I have two functions to run in parallel and each of them returns a value. I need to wait for both functions to finish and then process the returns from them. How could I achieve this in python. Assume

def fun1():
   #do some calculation#
   return val1

def fun2():
   #do some calculation#
   return val2

I want fun1 and fun2 to run in parallel and then need to calculate

valsum = val1+val2

Answer

falsetru picture falsetru · Jun 17, 2016

Using concurrent.futures:

from concurrent.futures import ProcessPoolExecutor as Executor
#from concurrent.futures import ThreadPoolExecutor as Executor  # to use threads
with Executor() as executor:
    future1 = executor.submit(fun1, arg1, arg2, ...)
    future2 = executor.submit(fun2, arg1, arg2, ...)
    val1, val2 = future1.result(), future2.result()
    valsum = val1 + val2

concurrent.futures.Executor.submit schedules the function to be executed, and returns a concurrent.futures.Future object.

concurrent.futures.Future.resultreturns the return value returned by the function.


Using multiprocessing.pool:

from multiprocessing.pool import Pool
#from multiprocessing.pool import ThreadPool as Pool  # to use threads
with multiprocessing.pool.Pool() as pool:
    result1 = pool.apply_async(fun1, (arg1, arg2, ...))
    result2 = pool.apply_async(fun2, (arg1, arg2, ...))
    val1, val2 = result1.get(), result2.get()
    valsum = val1 + val2

multiprocessing.pool.Pool.apply_async returns an AsyncResult object, and AsyncResult.get will return the return value of the function once it finish and the result arrive.