multiprocessing vs multithreading vs asyncio in Python 3

python multithreading python-3.x multiprocessing python-asyncio
user3654650 picture user3654650 · Dec 12, 2014 · Viewed 32.3k times · Source

I found that in Python 3.4 there are few different libraries for multiprocessing/threading: multiprocessing vs threading vs asyncio.

But I don't know which one to use or is the "recommended one". Do they do the same thing, or are different? If so, which one is used for what? I want to write a program that uses multicores in my computer. But I don't know which library I should learn.

Answer

user3159253 picture user3159253 · Dec 12, 2014

They are intended for (slightly) different purposes and/or requirements. CPython (a typical, mainline Python implementation) still has the global interpreter lock so a multi-threaded application (a standard way to implement parallel processing nowadays) is suboptimal. That's why multiprocessing may be preferred over threading. But not every problem may be effectively split into [almost independent] pieces, so there may be a need in heavy interprocess communications. That's why multiprocessing may not be preferred over threading in general.

asyncio (this technique is available not only in Python, other languages and/or frameworks also have it, e.g. Boost.ASIO) is a method to effectively handle a lot of I/O operations from many simultaneous sources w/o need of parallel code execution. So it's just a solution (a good one indeed!) for a particular task, not for parallel processing in general.

Related questions

Python 3: does Pool keep the original order of data passed to map?

I have written a little script to distribute workload between 4 threads and to test whether the results stay ordered (in respect to the order of the input): from multiprocessing import Pool import numpy as np import time import random rows = 16 …

Read More
On what CPU cores are my Python processes running?

The setup I have written a pretty complex piece of software in Python (on a Windows PC). My software starts basically two Python interpreter shells. The first shell starts up (I suppose) when you double click the main.py file. …

Read More
Are lists thread-safe?

I notice that it is often suggested to use queues with multiple threads, instead of lists and .pop(). Is this because lists are not thread-safe, or for some other reason?

Read More