How do I add valgrind tests to my cmake "test" target

Question

I run unit tests by building with ninja and then running ninja test from my build tree:

cmake -G Ninja /source/tree
ninja
ninja test

However, to run valgrind I need to run it manually:

valgrind rel/path/to/test

I'd like valgrind to run automatically when I run ninja test. According to the cmake documentation "setting up [valgrind tests] is extremely easy", but when I run

ctest -D NightlyMemoryCheck

I just get this error:

Cannot find file: /home/arman/tinman/deb/DartConfiguration.tcl
   Site: 
   Build name: (empty)
WARNING: No nightly start time found please set in CTestConfig.cmake or DartConfig.cmake
Problem initializing the dashboard.

I get a similar error when I follow the instructions from this SO question:

How do I make ctest run a program with valgrind without dart?

I don't know what dart is, but according the website it's some kind of online testing doodad.

Clearly extremely easy is not easy enough for me. Does anyone know of a solution that is so supremely easy that you'd have to be some kind of IT warlock to make it not work?

Answer 1

Here is a self contained example that shows how to add valgrind tests to a CMake project. The example consists of a single C++ source file main.cpp:

#include <iostream>

int main()
{
    double* leak = new double[10];
    std::cout << "Hello!" << std::endl;
}

The code contains an intentional leak which should be picked up by valgrind. We also need a CMakeLists.txt file which requires CMake >= 2.8:

cmake_minimum_required(VERSION 2.8)

project (ValgrindExample)

include (CTest)
add_executable(example main.cpp)
add_test(example_test example)

Here it is important to include the CTest module with include instead of just enabling tests with enable_testing(). The CTest module takes care of setting up the machinery for being able to run memory checks with tests (e.g., it locates the valgrind executable).

Now we can open a shell session in the project folder and create a Ninja build tree:

$ mkdir build; cd build
$ cmake -G Ninja -DCMAKE_BUILD_TYPE=Debug ..

We can build and run tests without valgrind in the regular way:

$ ninja
[2/2] Linking CXX executable example
$ ninja test
[0/1] Running tests...
...
100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.01 sec

To run tests with valgrind we have to use CMake's ctest executable with the test action memcheck:

$ ctest -T memcheck
...
1/1 MemCheck #1: example_test .....................   Passed    0.77 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.77 sec
-- Processing memory checking output: 
Memory checking results:
Memory Leak - 2

ctest prints a summary of the memory checking results. The detailed output of valgrind is located in a temporary directory in the build tree:

$ cat ./Testing/Temporary/MemoryChecker.*.log
==4565== 80 bytes in 1 blocks are definitely lost in loss record 37 of 64
==4565==    at 0x10000B681: malloc (in /usr/local/Cellar/valgrind/3.12.0/lib/valgrind/vgpreload_memcheck-amd64-darwin.so)
==4565==    by 0x1000507DD: operator new(unsigned long) (in /usr/lib/libc++.1.dylib)
==4565==    by 0x100000F93: main (main.cpp:5)
...

It is not possible to automatically run valgrind when you run ninja test because CMake's built-in test target cannot be modified and always runs tests in the regular way. We can however add a custom CMake target which invokes ctest with the -T memcheck option and then prints the detailed valgrind report:

add_custom_target(test_memcheck
    COMMAND ${CMAKE_CTEST_COMMAND} 
        --force-new-ctest-process --test-action memcheck
    COMMAND cat "${CMAKE_BINARY_DIR}/Testing/Temporary/MemoryChecker.*.log"
    WORKING_DIRECTORY "${CMAKE_BINARY_DIR}")

--test-action is the verbose version of the switch -T.

We then can invoke valgrind testing from Ninja with

$ ninja test_memcheck

and get the results as if we ran valgrind manually.