numpy and Global Interpreter Lock

python multithreading numpy gil python-multithreading
NPE picture NPE · Jun 1, 2011 · Viewed 8.3k times · Source

I am about to write some computationally-intensive Python code that'll almost certainly spend most of its time inside numpy's linear algebra functions.

The problem at hand is embarrassingly parallel. Long story short, the easiest way for me to take advantage of that would be by using multiple threads. The main barrier is almost certainly going to be the Global Interpreter Lock (GIL).

To help design this, it would be useful to have a mental model for which numpy operations can be expected to release the GIL for their duration. To this end, I'd appreciate any rules of thumb, dos and don'ts, pointers etc.

In case it matters, I'm using 64-bit Python 2.7.1 on Linux, with numpy 1.5.1 and scipy 0.9.0rc2, built with Intel MKL 10.3.1.

Answer

Ferdinand Beyer picture Ferdinand Beyer · Jun 1, 2011

You will probably find answers to all your questions regarding NumPy and parallel programming on the official wiki.

Also, have a look at this recipe page -- it contains example code on how to use NumPy with multiple threads.

Related questions

Python Exception in thread Thread-1 (most likely raised during interpreter shutdown)?

My friend and I have been working on a large project to learn and for fun in python and PyGame. Basically it is an AI simulation of a small village. we wanted a day/night cycle so I found a …

Read More
numpy OpenBLAS set maximum number of threads

I am using numpy and my model involves intensive matrix-matrix multiplication. To speed up, I use OpenBLAS multi-threaded library to parallelize the numpy.dot function. My setting is as follows, OS : CentOS 6.2 server #CPUs = 12, #MEM = 96GB python version: Python2.7.6 numpy : …

Read More
How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More