Python hangs in futex calls

python multithreading debugging deadlock futex
Helmut Grohne picture Helmut Grohne · Oct 11, 2010 · Viewed 7.1k times · Source

I have a Python daemon running in production. It employs between 7 and 120 threads. Recently the smallest instance (7 threads) started to show hangs while all other instances never showed this kind of problem. Attaching strace to the python process shows that all threads are calling futex FUTEX_WAIT_PRIVATE, so they are probably trying to lock something.

How would you debug such a problem?

Note that this is a production system running from flash memory, so disk writes are constrained, too.

Answer

Helmut Grohne picture Helmut Grohne · Oct 12, 2010

The observation was slightly incorrect. One thread wasn't calling futex, but instead swapping while holding the gil. Since the machine in question is low hardware this swapping took very long and seemed to be a deadlock. The underlying problem is a memory leak. :-(

Related questions

How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More
Is there any way to kill a Thread?

Is it possible to terminate a running thread without setting/checking any flags/semaphores/etc.?

Read More
Run certain code every n seconds

Is there a way to, for example, print Hello World! every n seconds? For example, the program would go through whatever code I had, then once it had been 5 seconds (with time.sleep()) it would execute that code. I would …

Read More