"chunksize" parameter in Python's multiprocessing.Pool.map

python multiprocessing
sergio picture sergio · Sep 29, 2010 · Viewed 36.8k times · Source

If I have a pool object with 2 processors for example:

p=multiprocessing.Pool(2)

and I want to iterate over a list of files on directory and use the map function

could someone explain what is the chunksize of this function:

p.map(func, iterable[, chunksize])

If I set the chunksize for example to 10 does that means every 10 files should be processed with one processor?

Answer

detly picture detly · Sep 29, 2010

Looking at the documentation for Pool.map it seems you're almost correct: the chunksize parameter will cause the iterable to be split into pieces of approximately that size, and each piece is submitted as a separate task.

So in your example, yes, map will take the first 10 (approximately), submit it as a task for a single processor... then the next 10 will be submitted as another task, and so on. Note that it doesn't mean that this will make the processors alternate every 10 files, it's quite possible that processor #1 ends up getting 1-10 AND 11-20, and processor #2 gets 21-30 and 31-40.

Related questions

How to use multiprocessing pool.map with multiple arguments?

In the Python multiprocessing library, is there a variant of pool.map which supports multiple arguments? text = "test" def harvester(text, case): X = case[0] text+ str(X) if __name__ == '__main__': pool = multiprocessing.Pool(processes=6) case = RAW_DATASET pool.…

Read More
Multiprocessing vs Threading Python

I am trying to understand the advantages of multiprocessing over threading. I know that multiprocessing gets around the Global Interpreter Lock, but what other advantages are there, and can threading not do the same thing?

Read More
multiprocessing.Pool: When to use apply, apply_async or map?

I have not seen clear examples with use-cases for Pool.apply, Pool.apply_async and Pool.map. I am mainly using Pool.map; what are the advantages of others?

Read More