doc/contrib/unit_tests.rst
######################## Adding and running tests ########################
A high-quality suite of tests is crucial in ensuring correctness and robustness of the codebase. Here, we provide instructions how to run unit tests, and also how to add a new one.
Contents
.. contents:: :backlinks: none :local:
Adding a new unit test
Add your test under the directories
tests/python/ <https://github.com/dmlc/xgboost/tree/master/tests/python>_tests/python-gpu/ <https://github.com/dmlc/xgboost/tree/master/tests/python-gpu>_ (if you are testing GPU code)tests/test_distributed <https://github.com/dmlc/xgboost/tree/master/tests/test_distributed>_. (if a distributed framework is used)Refer to the PyTest tutorial <https://docs.pytest.org/en/latest/getting-started.html>_
to learn how to write tests for Python code.
You may try running your test by following instructions in :ref:this section <running_pytest>.
Add your test under the directory tests/cpp/ <https://github.com/dmlc/xgboost/tree/master/tests/cpp>. Refer to this excellent tutorial on using Google Test <https://developer.ibm.com/articles/au-googletestingframework/>.
You may try running your test by following instructions in :ref:this section <running_gtest>. Note. Google Test version 1.8.1 or later is required.
The JVM packages for XGBoost (XGBoost4J / XGBoost4J-Spark) use the Maven Standard Directory Layout <https://maven.apache.org/guides/introduction/introduction-to-the-standard-directory-layout.html>_. Specifically, the tests for the JVM packages are located in the following locations:
jvm-packages/xgboost4j/src/test/ <https://github.com/dmlc/xgboost/tree/master/jvm-packages/xgboost4j/src/test>_jvm-packages/xgboost4j-spark/src/test/ <https://github.com/dmlc/xgboost/tree/master/jvm-packages/xgboost4j-spark/src/test>_To write a test for Java code, see JUnit 5 tutorial <https://junit.org/junit5/docs/current/user-guide/>.
To write a test for Scala, see Scalatest tutorial <http://www.scalatest.org/user_guide/writing_your_first_test>.
You may try running your test by following instructions in :ref:this section <running_jvm_tests>.
Add your test under the directory R-package/tests/testthat <https://github.com/dmlc/xgboost/tree/master/R-package/tests/testthat>. Refer to this excellent tutorial on testthat <https://kbroman.org/pkg_primer/pages/tests.html>.
You may try running your test by following instructions in :ref:this section <running_r_tests>.
Running Unit Tests Locally
.. _running_r_tests:
Run
.. code-block:: bash
python ./ops/script/test_r_package.py --task=check
at the root of the project directory. The command builds and checks the XGBoost r-package. Alternatively, if you want to just run the tests, you can use the following commands after installing XGBoost:
.. code-block:: bash
cd R-package/tests/ Rscript testthat.R
.. _running_jvm_tests:
Maven is used
.. code-block:: bash
mvn test
.. _running_pytest:
To run Python unit tests, first install pytest <https://docs.pytest.org/en/latest/contents.html>_ package:
.. code:: bash
pip3 install pytest
Then compile XGBoost according to instructions in :ref:build_shared_lib. Finally, invoke pytest at the project root directory:
.. code:: bash
export PYTHONPATH=./python-package pytest -v -s --fulltrace tests/python
In addition, to test CUDA code, run:
.. code:: bash
export PYTHONPATH=./python-package pytest -v -s --fulltrace tests/python-gpu
(For this step, you should have compiled XGBoost with CUDA enabled.)
For testing with distributed frameworks like Dask and PySpark:
.. code:: bash
export PYTHONPATH=./python-package pytest -v -s --fulltrace tests/test_distributed
.. _running_gtest:
To build and run C++ unit tests enable tests while running CMake:
.. code-block:: bash
cmake -B build -S . -GNinja -DGOOGLE_TEST=ON -DUSE_DMLC_GTEST=ON -DUSE_CUDA=ON -DUSE_NCCL=ON cmake --build build cd ./build ./testxgboost
Flags like USE_CUDA, USE_DMLC_GTEST are optional. For more info about how to build
XGBoost from source, see :doc:/build. One can also run all unit tests using ctest tool
which provides higher flexibility. For example:
.. code-block:: bash
ctest --verbose
If you need to debug errors on Windows using the debugger from VS, you can append the gtest flags in test_main.cc:
.. code-block::
::testing::GTEST_FLAG(filter) = "Suite.Test"; ::testing::GTEST_FLAG(repeat) = 10;
Sanitizers: Detect memory errors and data races
By default, sanitizers are bundled in GCC and Clang/LLVM. One can enable sanitizers with GCC >= 4.8 or LLVM >= 3.1, But some distributions might package sanitizers separately. Here is a list of supported sanitizers with corresponding library names:
Memory sanitizer is exclusive to LLVM, hence not supported in XGBoost. With latest compilers like gcc-9, when sanitizer flags are specified, the compiler driver should be able to link the runtime libraries automatically.
One can build XGBoost with sanitizer support by specifying -DUSE_SANITIZER=ON. By default, address sanitizer and leak sanitizer are used when you turn the USE_SANITIZER flag on. You can always change the default by providing a semicolon separated list of sanitizers to ENABLED_SANITIZERS. Note that thread sanitizer is not compatible with the other two sanitizers.
.. code-block:: bash
cmake -DUSE_SANITIZER=ON -DENABLED_SANITIZERS="address;undefined" /path/to/xgboost
By default, CMake will search regular system paths for sanitizers, you can also supply a specified SANITIZER_PATH.
.. code-block:: bash
cmake -DUSE_SANITIZER=ON -DENABLED_SANITIZERS="address;undefined"
-DSANITIZER_PATH=/path/to/sanitizers /path/to/xgboost
Running XGBoost on CUDA with address sanitizer (asan) will raise memory error. To use asan with CUDA correctly, you need to configure asan via ASAN_OPTIONS environment variable:
.. code-block:: bash
ASAN_OPTIONS=protect_shadow_gap=0 ${BUILD_DIR}/testxgboost
By default undefined sanitizer doesn't print out the backtrace. You can enable it by exporting environment variable:
.. code-block::
UBSAN_OPTIONS=print_stacktrace=1 ${BUILD_DIR}/testxgboost
For details, please consult official documentation <https://github.com/google/sanitizers/wiki>_ for sanitizers.