Help/dev/testing.rst
CMake Testing Guide
The following is a guide to the CMake test suite for developers.
See documentation on CMake Development_ for more information.
See CMake Integration Testing_ for running integration testing builds.
See Tests/README.rst_ for the test suite layout in the source tree.
.. _CMake Development: README.rst
.. _CMake Integration Testing: integration-testing.rst
.. _Tests/README.rst: ../../Tests/README.rst
After Building CMake, one may run the test suite in the build tree
using ctest(1):
With a single-configuration CMake generator, such as Ninja
or Unix Makefiles, one may simply run ctest:
.. code-block:: console
$ ctest
With a multi-configuration CMake generator, such as
:generator:Ninja Multi-Config, :generator:Visual Studio <Visual Studio Generators>, or :generator:Xcode, one must tell ctest which
configuration to test by passing the -C <config> option:
.. code-block:: console
$ ctest -C Debug
Some useful ctest(1)_ options include:
-N
List test names without running them.
-V
Show verbose output from each test.
-j <N>
Run to run up to N tests concurrently.
-R <regex>
Select tests for which the regular expression matches a substring
of their name.
Many CMake tests create their own test project build trees underneath
the Tests/ directory at the top of the CMake build tree. These
build trees are left behind after testing completes in order to
facilitate manual investigation of results. Many of the tests do not
clean their build trees if they are run again, with the exception of
tests using the RunCMake_ infrastructure.
In order to clear test build trees, drive the test_clean custom target
in the CMake build tree:
.. code-block:: console
$ cmake --build . --target test_clean
This removes the Tests/ subdirectories created by individual tests
so they will use a fresh directory next time they run.
.. _Building CMake: ../../README.rst#building-cmake
.. _ctest(1): https://cmake.org/cmake/help/latest/manual/ctest.1.html
.. _RunCMake: ../../Tests/RunCMake/README.rst
After Building CMake_ with one CMake generator, one may configure the
test suite using a different generator in a separate build tree, without
building CMake itself again, by defining CMake_TEST_EXTERNAL_CMAKE
to be the absolute path to the bin directory containing the cmake,
ctest, and cpack executables.
For example, after building CMake with the Ninja generator:
.. code-block:: console
$ cmake -B build-ninja -G Ninja -DCMAKE_BUILD_TYPE=Debug $ cmake --build build-ninja
one may configure a second build tree to drive tests with the
Ninja Multi-Config generator:
.. code-block:: console
$ cmake -B build-nmc-tests -G "Ninja Multi-Config"
-DCMake_TEST_EXTERNAL_CMAKE="$PWD/build-ninja/bin"
$ cmake --build build-nmc-tests --config Release
The second build tree does not build CMake itself, but does configure the test suite and build test binaries. One may then run tests normally:
.. code-block:: console
$ cd build-nmc-tests $ ctest -C Release
Note that the configuration with which one drives tests in the second build tree is independent of the configuration with which CMake was built in the first.