How to debug a rare deadlock?

c++ multithreading c++11 threadpool deadlock
Lærne picture Lærne · Jan 4, 2015 · Viewed 8.2k times · Source

I'm trying to debug a custom thread pool implementation that has rarely deadlocks. So I cannot use a debugger like gdb because I have click like 100 times "launch" debugger before having a deadlock.

Currently, I'm running the threadpool test in an infinite loop in a shell script, but that means I cannot see variables and so on. I'm trying to std::cout data, but that slow down the thread and reduce the risk of deadlocks meaning that I can wait like 1hour with my infinite before getting messages. Then I don't get the error, and I need more messages, which means waiting one more hour...

How to efficiently debug the program so that its restart over and over until it deadlocks ? (Or maybe should I open another question with all the code for some help ?)

Thank you in advance !

Bonus question : how to check everything goes fine with a std::condition_variable ? You cannot really tell which thread are asleep or if a race condition occurs on the wait condition.

Answer

Maciej Piechotka picture Maciej Piechotka · Jan 4, 2015

There are 2 basic ways:

  1. Automate the running of program under debugger. Using gdb program -ex 'run <args>' -ex 'quit' should run the program under debugger and then quit. If the program is still alive in one form or another (segfault, or you broke it manually) you will be asked for confirmation.
  2. Attach the debugger after reproducing the deadlock. For example gdb can be run as gdb <program> <pid> to attach to running program - just wait for deadlock and attach then. This is especially useful when attached debugger causes timing to be changed and you can no longer repro the bug.

In this way you can just run it in loop and wait for result while you drink coffee. BTW - I find the second option easier.

Related questions

Thread pooling in C++11

Relevant questions: About C++11: C++11: std::thread pooled? Will async(launch::async) in C++11 make thread pools obsolete for avoiding expensive thread creation? About Boost: C++ boost thread reusing threads boost::thread and creating a pool of them! How do …

Read More
Does async(launch::async) in C++11 make thread pools obsolete for avoiding expensive thread creation?

It is loosely related to this question: Are std::thread pooled in C++11?. Though the question differs, the intention is the same: Question 1: Does it still make sense to use your own (or 3rd-party library) thread pools to avoid expensive …

Read More
Start thread with member function

I am trying to construct a std::thread with a member function that takes no arguments and returns void. I can't figure out any syntax that works - the compiler complains no matter what. What is the correct way to …

Read More