What is the difference between glibc's MALLOC_CHECK_, M_CHECK_ACTION, and mcheck?

Question

glibc seems to have more than one way of doing some heap checking:

1. mallopt with the M_CHECK_ACTION parameter
2. the MALLOC_CHECK_ environment variable
3. the mcheck family of functions

I find the available documentation to be confusing. The manual doesn't list M_CHECK_ACTION at all when describing mallopt. This mallopt man page, however, does describe M_CHECK_ACTION. Additionally, it says it's equivalent to the environment variable MALLOC_CHECK_:

   MALLOC_CHECK_
          This environment variable controls the same parameter as
          mallopt() M_CHECK_ACTION.  If this variable is set to a
          nonzero value, then a special implementation of the memory-
          allocation functions is used.  (This is accomplished using the
          malloc_hook(3) feature.)  This implementation performs
          additional error checking, but is slower than the standard set
          of memory-allocation functions.

The glibc manual has a page for mcheck and friends and describes them as "heap consistency checking". It is on this page that the manual discusses MALLOC_CHECK_:

Another possibility to check for and guard against bugs in the use of malloc, realloc and free is to set the environment variable MALLOC_CHECK_. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free with the same argument, or overruns of a single byte (off-by-one bugs).

So mcheck et al is an alternative to MALLOC_CHECK_/M_CHECK_ACTION?

Furthermore, how does one disable all this super-helpful consistency checking? The man page says setting MALLOC_CHECK_ (and therefore M_CHECK_ACTION) to 0 will not use "a special implementation of the memory-allocation functions." The glibc manual, however, states that "When MALLOC_CHECK_ is set, a special (less efficient) implementation is used." An environment variable set to 0 is still set, so one of these is wrong.

My own experiments (using the example program from this mcheck man page) show that not having MALLOC_CHECK_ set at all results in the same behavior as MALLOC_CHECK_=3 (on RHEL 6.4). And mcheck seems completely unrelated, as it can be set independently.

Answer 1

There are four different consistency checks. The following corresponds to glibc 2.25.

1. Various asserts. Whether they are active is determined when glibc is built. In the past, Debian and downstreams left asserts switched on. Fedora and downstreams disabled them in the past (but current Fedora leaves them on).
2. Consistency checks in the core malloc implementation which do not use asserts, but malloc_printerr. In the past, they could be switched off using MALLOC_CHECK_=0 (and mallopt). However, the I doubt that the error recovery after malloc_printerr is correct in all cases, so this is unlikely to work. And if there is heap corruption, the program might crash soon anyway.
3. Relatively lightweight heap buffer overflow detection. This is enabled by MALLOC_CHECK_=3 (or another non-zero value). This cannot be switched on by mallopt. It is implemented by increasing the size of allocations and storing some padding and checking it in some places. This heap checker should be thread-safe, and it is tightly coupled with malloc internals. However, it is rarely used, so there could easily be annoying bugs.
4. The mcheck function, called from __malloc_initialize_hook by linking with libmcheck.a. This is completely different from the previous checks. I think the idea is to verify correct allocation behavior by having a separate set of metadata (independent of the actual allocator), and mcheck does not depend on the glibc malloc internals except for the malloc hooks. However, its use of these hooks completely not thread safe. I don't think anyone uses mcheck today. It is just included because no one has removed it yet. (It is not even needed for backwards compatibility because there is no guarantee that all heap corruption is detected, and applications which corrupt the heap are still completely undefined, so there is no way to detect whether mcheck actually performs additional checks.)

In addition to that, there is also MALLOC_PERTURB_, which can be used to detect some forms of access to uninitialized or freed data.