Can't pickle Function

python multiprocessing pool
Casey Daniel picture Casey Daniel · Mar 3, 2014 · Viewed 18.6k times · Source

So I'm trying to speed up my computation time by doing a little bit multiprocessing

I'm trying to use the pool workers.

At the top of my code I have 

import Singal as s
import multiprocessing as mp
def wrapper(Channel):
    Noise_Frequincies = []
    for i in range(1,125):
        Noise_Frequincies.append(60.0*float(i))
    Noise_Frequincies.append(180.0)
    filter1 = s.Noise_Reduction(Sample_Rate,Noise_Frequincies,Channel)
    return filter1

Then when the time comes I use

Both_Channels = [Chan1, Chan2]
results = mp.Pool(2).map(wrapper,Both_Channels)
filter1 = results[0]
filter2 = results[1]

I get the following error 

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 808, in __bootstrap_inner
self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 761, in run
self.__target(*self.__args, **self.__kwargs)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 342, in _handle_tasks
put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

Chan1 and Chan2 are arrays of my signal, and I'm trying to filter out some noise out of each. I'm new to multiprocessing so I apologize if this is a dumb error

Answer

zmo picture zmo · Mar 3, 2014

I marked this question as dup of that Q/A, and to sum up: you can't pickle functions, and that's what you're trying to do when you pass wrapper() to Pool().map(). To pickle functions, you need to use copy_reg, as shown by this example.

" Classes, functions, and methods cannot be pickled -- if you pickle an object, the object's class is not pickled, just a string that identifies what class it belongs to. " (cf doc)

I don't use custom classes, and that's where his issue is (at least as far as I can tell)

no, his problem that he was trying to pickle an instance method, which is relatively close to a function, as both can't be pickled. And the workaround for that A should work for you too.

Though I did not test it…

HTH

Related questions

Can't pickle <type 'instancemethod'> when using multiprocessing Pool.map()

I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine: import multiprocessing def f(x): return x*x def go(): pool = multiprocessing.Pool(processes=4) print pool.map(f, …

Read More
Passing multiple parameters to pool.map() function in Python

I need some way to use a function within pool.map() that accepts more than one parameter. As per my understanding, the target function of pool.map() can only have one iterable as a parameter but is there a way …

Read More
How to troubleshoot an "AttributeError: __exit__" in multiproccesing in Python?

I tried to rewrite some csv-reading code to be able to run it on multiple cores in Python 3.2.2. I tried to use the Pool object of multiprocessing, which I adapted from working examples (and already worked for me for another …

Read More