Why changing start method to 'spawn' from 'fork' in Python multiprocessing does not allow me run my job anymore?

python python-3.x multiprocessing python-multiprocessing spawn
Amir picture Amir · Apr 3, 2018 · Viewed 11.4k times · Source

I am able to run a background function using multiprocessing.Process with the start method fork. For some reason, I need this child process to start a new environment when running. So I set the start method to spawn via multiprocessing.set_start_method('spawn') and run the job via job.start() I get the following error:

Can't pickle <class 'module'>: attribute lookup module on builtins failed

However, I do not use pickle for anything within the function that I am calling. What could I be doing wrong? Is there a general rule of thumb that I should have followed when running processes in spawn mode?

FYI: I am on a machine with Ubuntu 16.04

Answer

J_H picture J_H · Apr 3, 2018

Is there a general rule of thumb...

Yes. You ran into this documented restriction:

https://docs.python.org/3/library/multiprocessing.html

There are a few extra restriction which don’t apply to the fork start method.

More picklability

Ensure that all arguments to Process.init() are picklable. Also, if you subclass Process then make sure that instances will be picklable when the Process.start method is called.

You are running on ubuntu, so fork is probably the right answer. If there is a requirement you need to address which fork is incompatible with, then you will want to clearly document the details as the first part of choosing an improved solution.

Related questions

What's the difference between ThreadPool vs Pool in the multiprocessing module?

Whats the difference between ThreadPool and Pool in multiprocessing module. When I try my code out, this is the main difference I see: from multiprocessing import Pool import os, time print("hi outside of main()") def hello(x): print("inside …

Read More
Python multiprocessing: understanding logic behind `chunksize`

What factors determine an optimal chunksize argument to methods like multiprocessing.Pool.map()? The .map() method seems to use an arbitrary heuristic for its default chunksize (explained below); what motivates that choice and is there a more thoughtful approach based …

Read More
How to troubleshoot an "AttributeError: __exit__" in multiproccesing in Python?

I tried to rewrite some csv-reading code to be able to run it on multiple cores in Python 3.2.2. I tried to use the Pool object of multiprocessing, which I adapted from working examples (and already worked for me for another …

Read More