Investigation: C++ Unit Test Error Propagation

Issue

The user reported that C++ unit tests, when run with bash test --cpp --clean, were not propagating errors up to test.py, often showing "0/1 runs" despite an underlying failure.

Investigation Steps and Findings

1. Analyzed test script: The test script (located at C:\Users\niteris\dev\fastled10\test) is a simple bash script that executes uv run test.py "$@". This means all arguments, including --cpp and --clean, are passed directly to test.py.

2. Analyzed test.py:

   • test.py is the main test runner.
   • It uses ci.util.test_runner.runner to orchestrate test execution.
   • It includes a try...except SystemExit block in its main function to catch failures and set a tests_passed flag, ultimately exiting with a non-zero status code if tests_passed is False.

3. Analyzed ci/util/test_runner.py:

   • The runner function in this file determines test categories and, for C++ unit tests, delegates to ci.util.meson_runner.run_meson_build_and_test.
   • It expects run_meson_build_and_test to return a MesonTestResult object, and if result.success is False, it calls sys.exit(1).

4. Analyzed ci/util/meson_runner.py:

   • This file is responsible for building and running C++ unit tests using the Meson build system.
   • The run_meson_build_and_test function orchestrates the process:
     • It handles the --clean flag by removing the build directory, forcing a full rebuild.
     • It calls setup_meson_build for Meson configuration.
     • It calls compile_meson for compilation. This function uses RunningProcess to execute meson compile and checks its returncode. If compilation fails (non-zero returncode), it returns False.
     • It calls run_meson_test for test execution. This function uses RunningProcess to execute meson test, parses its output to count passed/failed tests, and returns MesonTestResult(success=False, ...) if tests fail or meson test exits with a non-zero code.
   • Crucially, if any of these sub-steps fail, run_meson_build_and_test returns a MesonTestResult with success=False.

5. Injected a Failing Test: To confirm error propagation, a simple failing TEST_CASE (CHECK(1 == 2);) was temporarily added to tests/doctest_main.cpp.

6. Executed bash test --cpp --clean with Failing Test:

   • The command was executed, forcing a clean build and re-run of C++ tests.
   • The output clearly showed:
     • Meson setup and compilation proceeded successfully.
     • The doctest_main.cpp file was compiled.
     • The [MESON] Streaming test execution complete: section reported:
       • Tests run: 1
       • Passed: 0
       • Failed: 1
     • The overall summary indicated: [MESON] ❌ Some tests failed (0/1 tests in X.XXs).
     • The run_shell_command tool reported Exit Code: 1.

Conclusion

The investigation confirms that the error is propagating correctly from the C++ unit test (via doctest) through the Meson build system, ci/util/meson_runner.py, ci/util/test_runner.py, and finally to test.py, which then exits with a non-zero status code (1) to the shell.

The initial observation of "0/1 runs" was likely due to:

• The specific C++ test not failing in a way that meson test could categorize as a "run" (e.g., a crash before test execution could begin).
• The test being skipped due to fingerprint caching (which the --clean flag successfully bypasses).
• A misinterpretation of partial output.

The current test infrastructure correctly identifies and propagates C++ unit test failures.

Remediation

The temporary failing test case in tests/doctest_main.cpp has been removed.