RabbitMQ install issue on Centos 5.5

Question

I've been trying to get rabbitmq-server-2.4.0 up and running on Centos 5.5 on an Amazon AWS instance.

My instance uses the following kernel: 2.6.18-xenU-ec2-v1.2

I've tried installation of erlang and rabbitmq-server using: 1) yum repos 2) direct rpm installation 3) compiling from source.

In every case, I get the following message when attempting to start the RabbitMQ-Server process:

pthread/ethr_event.c:98: Fatal error in wait__(): Function not implemented (38)

Any help would be appreciated.

Answer 1

The problem

When starting erlang, the message pthread/ethr_event.c:98: Fatal error in wait__(): Function not implemented (38) is, on modern distros, most likely the result of a precompiled Erlang binary interacting with a kernel that doesn't implement FUTEX_WAIT_PRIVATE and FUTEX_WAKE_PRIVATE. The kernels Amazon provides for EC2 do not implement these FUTEX_PRIVATE_ macros.

Attempting to build Erlang from source on an ec2 box may fail in the same way if the distro installs kernel headers into /usr/include/linux as a requirement of other packages. (E.g., Centos requires the kernel-headers package as a prerequisite for gcc, gcc-c++, glibc-devel and glibc-headers, among others). Since the headers installed by the package do not match the kernel installed by the EC2 image creation scripts, Erlang incorrectly assumes FUTEX_WAIT_PRIVATE and FUTEX_WAKE_PRIVATE are available.

The fix

To fix it, the fastest is to manually patch erts/include/internal/pthread/ethr_event.h to use the non-_PRIVATE futex implementation:

#if defined(FUTEX_WAIT_PRIVATE) && defined(FUTEX_WAKE_PRIVATE)
#  define ETHR_FUTEX_WAIT__ FUTEX_WAIT_PRIVATE
#  define ETHR_FUTEX_WAKE__ FUTEX_WAKE_PRIVATE
#else
#  define ETHR_FUTEX_WAIT__ FUTEX_WAIT
#  define ETHR_FUTEX_WAKE__ FUTEX_WAKE
#endif

should become

//#if defined(FUTEX_WAIT_PRIVATE) && defined(FUTEX_WAKE_PRIVATE)
//#  define ETHR_FUTEX_WAIT__ FUTEX_WAIT_PRIVATE
//#  define ETHR_FUTEX_WAKE__ FUTEX_WAKE_PRIVATE  
//#else
#  define ETHR_FUTEX_WAIT__ FUTEX_WAIT
#  define ETHR_FUTEX_WAKE__ FUTEX_WAKE
//#endif

Quick test

If you suspect the private futex issue is your problem, but want to verify it before you recompile all of Erlang, the following program can pin it down:

#include <sys/syscall.h>
#include <unistd.h>
#include <sys/time.h>
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <stdint.h>
typedef uint32_t u32; /* required on older kernel headers to fix a bug in futex.h Delete this line if it causes problems. */
#include <linux/futex.h>

int main(int argc, char *argv[])
{
#if defined(FUTEX_WAIT) && defined(FUTEX_WAKE) 
        uint32_t i = 1;
        int res = 0;
        res = syscall(__NR_futex, (void *) &i, FUTEX_WAKE, 1,
                        (void*)0,(void*)0, 0);
        if (res != 0)
        {
                printf("FUTEX_WAKE HAD ERR %i: %s\n", errno, strerror(errno));
        } else {
                printf("FUTEX_WAKE SUCCESS\n");
        }

        res = syscall(__NR_futex, (void *) &i, FUTEX_WAIT, 0,
                        (void*)0,(void*)0, 0);
        if (res != 0)
        {
                printf("FUTEX_WAIT HAD ERR %i: %s\n", errno, strerror(errno));
        } else {
                printf("FUTEX_WAIT SUCCESS\n");
        }
#else
        printf("FUTEX_WAKE and FUTEX_WAIT are not defined.\n");
#endif

#if defined(FUTEX_WAIT_PRIVATE) && defined(FUTEX_WAKE_PRIVATE) 
        uint32_t j = 1;
        int res_priv = 0;
        res_priv = syscall(__NR_futex, (void *) &j, FUTEX_WAKE_PRIVATE, 1,
                        (void*)0,(void*)0, 0);
        if (res_priv != 0)
        {
                printf("FUTEX_WAKE_PRIVATE HAD ERR %i: %s\n", errno, strerror(errno));
        } else {
                printf("FUTEX_WAKE_PRIVATE SUCCESS\n");
        }

        res_priv = syscall(__NR_futex, (void *) &j, FUTEX_WAIT_PRIVATE, 0,
                        (void*)0,(void*)0, 0);
        if (res_priv != 0)
        {
                printf("FUTEX_WAIT_PRIVATE HAD ERR %i: %s\n", errno, strerror(errno));
        } else {
                printf("FUTEX_WAIT_PRIVATE SUCCESS\n");
        }
#else
        printf("FUTEX_WAKE_PRIVATE and FUTEX_WAIT_PRIVATE are not defined.\n");
#endif


        return 0;
}

Paste it into futextest.c, then gcc futextest.c and ./a.out.

If your kernel implements private futexes, you'll see

FUTEX_WAKE SUCCESS
FUTEX_WAIT SUCCESS
FUTEX_WAKE_PRIVATE SUCCESS
FUTEX_WAIT_PRIVATE SUCCESS

If you have a kernel without the _PRIVATE futex functions, you'll see

FUTEX_WAKE SUCCESS
FUTEX WAIT SUCCESS
FUTEX_WAKE_PRIVATE HAD ERR 38: Function not implemented
FUTEX_WAIT_PRIVATE HAD ERR 38: Function not implemented

This fix should allow Erlang to compile, and will yield an environment you can install rabbitmq against using the --nodeps method discussed here.