multiprocess.apply_async How do I wrap *args and **kwargs?

Question

I'm trying to get multiprocess.apply_async to take in both *args and **kwargs. The docs indicate that this might be possible with the calling sequence:

apply_async(func[, args[, kwds[, callback]]])

But I can not figure out how to get the calling syntax correct. With the minimal example:

from multiprocessing import Pool

def f(x, *args, **kwargs):
    print x, args, kwargs

args, kw = (), {}

print "# Normal call"
f(0, *args, **kw)

print "# Multicall"
P = Pool()
sol = [P.apply_async(f, (x,), *args, **kw) for x in range(2)]
P.close()
P.join()

for s in sol: s.get()

This works as expected giving the output

# Normal call
0 () {}
# Multicall
0 () {}
1 () {}

When args is not an empty tuple, for example args = (1,2,3), the single call works, but the multiprocessing solution gives:

# Normal call
0 (1, 2, 3) {}
# Multicall
Traceback (most recent call last):
  File "kw.py", line 16, in <module>
    sol = [P.apply_async(f, (x,), *args, **kw) for x in range(2)]
TypeError: apply_async() takes at most 5 arguments (6 given)

With the kwargs argument I get, for example kw = {'cat':'dog'}

# Normal call
0 () {'cat': 'dog'}
# Multicall
Traceback (most recent call last):
  File "kw.py", line 15, in <module>
    sol = [P.apply_async(f, (x,), *args, **kw) for x in range(2)]
TypeError: apply_async() got an unexpected keyword argument 'cat'

How do I properly wrap multiprocess.apply_async?

Answer 1

You don't have to use * and ** explicitly. Simply pass the tuple and the dict and let apply_async unpack them:

from multiprocessing import Pool

def f(x, *args, **kwargs):
    print x, args, kwargs

args, kw = (1,2,3), {'cat': 'dog'}

print "# Normal call"
f(0, *args, **kw)

print "# Multicall"
P = Pool()
sol = [P.apply_async(f, (x,) + args, kw) for x in range(2)]
P.close()
P.join()

for s in sol: s.get()

Output:

# Normal call                                                                                        
0 (1, 2, 3) {'cat': 'dog'}
# Multicall
0 (1, 2, 3) {'cat': 'dog'}
1 (1, 2, 3) {'cat': 'dog'}

Remember that in python's documentation, if a function accepts *args and **kwargs its signature explicitly states that:

the_function(a,b,c,d, *args, **kwargs)

In your case:

apply_async(func[, args[, kwds[, callback]]])

There are no * there, hence args is one argument, which is unpacked when calling func and kwargs is one argument and processed in the same way. Also note that it's not possible to have other arguments after **kwargs:

>>> def test(**kwargs, something=True): pass

  File "<stdin>", line 1
    def test(**kwargs, something=True): pass
                     ^
SyntaxError: invalid syntax