Python multiprocessing's Pool process limit

python multiprocessing cpu-cores
rottentomato56 picture rottentomato56 · Nov 18, 2013 · Viewed 52.7k times · Source

In using the Pool object from the multiprocessing module, is the number of processes limited by the number of CPU cores? E.g. if I have 4 cores, even if I create a Pool with 8 processes, only 4 will be running at one time?

Answer

Tim Peters picture Tim Peters · Nov 18, 2013

You can ask for as many processes as you like. Any limit that may exist will be imposed by your operating system, not by multiprocessing. For example,

 p = multiprocessing.Pool(1000000)

is likely to suffer an ugly death on any machine. I'm trying it on my box as I type this, and the OS is grinding my disk to dust swapping out RAM madly - finally killed it after it had created about 3000 processes ;-)

As to how many will run "at one time", Python has no say in that. It depends on:

  1. How many your hardware is capable of running simultaneously; and,
  2. How your operating system decides to give hardware resources to all the processes on your machine currently running.

For CPU-bound tasks, it doesn't make sense to create more Pool processes than you have cores to run them on. If you're trying to use your machine for other things too, then you should create fewer processes than cores.

For I/O-bound tasks, it may make sense to create a quite a few more Pool processes than cores, since the processes will probably spend most their time blocked (waiting for I/O to complete).

Related questions

How to use multiprocessing pool.map with multiple arguments?

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments? text = "test" def harvester(text, case): X = case[0] text+ str(X) if __name__ == '__main__': pool = multiprocessing.Pool(processes=6) case = RAW_DATASET pool.…

Read More
Multiprocessing vs Threading Python

I am trying to understand the advantages of multiprocessing over threading. I know that multiprocessing gets around the Global Interpreter Lock, but what other advantages are there, and can threading not do the same thing?

Read More
multiprocessing.Pool: When to use apply, apply_async or map?

I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map; what are the advantages of others?

Read More