Mutex in shared memory when one user crashes?
Suppose that a process is creating a mutex in shared memory and locking it and dumps core while the mutex is locked.
Now in another process how do I detect that mutex is already locked but not owned by any process?
Answer
It seems that the exact answer has been provided in the form of robust mutexes.
According to POSIX, pthread mutexes can be initialised "robust" using pthread_mutexattr_setrobust(). If a process holding the mutex then dies, the next thread to acquire it will receive EOWNERDEAD (but still acquire the mutex successfully) so that it knows to perform any cleanup. It then needs to notify that the acquired mutex is again consistent using pthread_mutex_consistent().
Obviously you need both kernel and libc support for this to work. On Linux the kernel support behind this is called "robust futexes", and I've found references to userspace updates being applied to glibc HEAD.
In practice, support for this doesn't seem to have filtered down yet, in the Linux world at least. If these functions aren't available, you might find pthread_mutexattr_setrobust_np() there instead, which as far as I can gather appears to be a non-POSIX predecessor providing the same semantics. I've found references to pthread_mutexattr_setrobust_np() both in Solaris documentation and in /usr/include/pthread.h on Debian.
The POSIX spec can be found here: http://www.opengroup.org/onlinepubs/9699919799/functions/pthread_mutexattr_setrobust.html