multiprocess or threading in python?

python multithreading multiprocess
Ryan picture Ryan · Aug 4, 2009 · Viewed 18.3k times · Source

I have a python application that grabs a collection of data and for each piece of data in that collection it performs a task. The task takes some time to complete as there is a delay involved. Because of this delay, I don't want each piece of data to perform the task subsequently, I want them to all happen in parallel. Should I be using multiprocess? or threading for this operation?

I attempted to use threading but had some trouble, often some of the tasks would never actually fire.

Answer

Christopher picture Christopher · Aug 4, 2009

If you are truly compute bound, using the multiprocessing module is probably the lightest weight solution (in terms of both memory consumption and implementation difficulty.)

If you are I/O bound, using the threading module will usually give you good results. Make sure that you use thread safe storage (like the Queue) to hand data to your threads. Or else hand them a single piece of data that is unique to them when they are spawned.

PyPy is focused on performance. It has a number of features that can help with compute-bound processing. They also have support for Software Transactional Memory, although that is not yet production quality. The promise is that you can use simpler parallel or concurrent mechanisms than multiprocessing (which has some awkward requirements.)

Stackless Python is also a nice idea. Stackless has portability issues as indicated above. Unladen Swallow was promising, but is now defunct. Pyston is another (unfinished) Python implementation focusing on speed. It is taking an approach different to PyPy, which may yield better (or just different) speedups.

Related questions

Python, using multiprocess is slower than not using it

After spending a lot of time trying to wrap my head around multiprocessing I came up with this code which is a benchmark test: Example 1: from multiprocessing import Process class Alter(Process): def __init__(self, word): Process.__init__(self) self.…

Read More
Multiprocess multiple files in a list

I am trying to read a list that contains N number of .csv files stored in a list synchronously. Right now I do the following: import multiprocess Empty list Append list with listdir of .csv's def A() -- even files (…

Read More
How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More