.. _emscripten-test-suite:

===================== Emscripten Test Suite

Emscripten has a comprehensive test suite, which covers virtually all Emscripten functionality. These tests are an excellent resource for developers as they provide practical examples of most features, and are known to pass on the main branch. In addition to correctness tests, there are also benchmarks that you can run.

This article explains how to run the test and benchmark suite, and provides an overview of what tests are available.

Setting up

To run the tests, you need an emscripten setup, as it will run emcc and other commands. See the :ref:developer's guide <developers-guide-setting-up> for how best to do that.

Running tests

Run the test suite runner (test/runner <https://github.com/emscripten-core/emscripten/blob/main/test/runner.py>_) with --help to see the help message:

.. code-block:: bash

test/runner --help

The tests are divided into modes. You can run either an entire mode or an individual test, or use wildcards to run some tests in some modes. For example:

.. code-block:: bash

run one test (in the default mode)

test/runner test_foo

run a bunch of tests in one mode (here, all i64 tests in -O3)

test/runner core3.test_i64

run all tests in a specific mode (here, wasm2gs -O1)

test/runner wasm2js1

The core test modes (defined at the bottom of test/test_core.py <https://github.com/emscripten-core/emscripten/blob/main/test/test_core.py>_) let you run the tests in variety of different configurations and with different optimization flags. For example, wasm2js or wasm64. There are also non-core test suites, that run tests in more special manner (in particular, in those tests it is not possible to say "run the test with a different optimization flag" - that is what the core tests are for). The non-core test suites include

other: Non-core tests running in the shell.
browser: Tests that run in a browser.
sockets: Networking tests that run in a browser.
interactive: Browser tests that are not fully automated, and require user interaction (these should be automated eventually).
sanity: Tests for emscripten setting itself up. This modifies your .emscripten file temporarily.
benchmark: Runs benchmarks, measuring speed and code size.

The wildcards we mentioned above work for non-core test modes too, for example:

.. code-block:: bash

run one browser test

test/runner browser.test_sdl_image

run all SDL2 browser tests

test/runner browser.test_sdl2*

run all browser tests

test/runner browser

Skipping Tests

An individual test can be skipped by passing the "skip:" prefix. E.g.

.. code-block:: bash

test/runner other skip:other.test_cmake

Wildcards can also be passed in skip, so

.. code-block:: bash

test/runner browser skip:browser.test_pthread_*

will run the whole browser suite except for all the pthread tests in it.

Exiting on first failure

Sometimes it is useful to be able to iteratively fix one test at a time. In this case the --failfast option can be used to exit the test runner after the first failure.

.. note:: This option only works with the serial test runner. For test suites that are normally run in parallel you can force them to run serially using -j1.

Once a test is fixed, you continue where you left off using --start-at option:

.. code-block:: bash

test/runner browser --start-at test_foo --failfast

Running a bunch of random tests

You can run a random subset of the test suite, using something like

.. code-block:: bash

test/runner random100

Replace 100 with another number as you prefer. This will run that number of random tests, and tell you the statistical likelihood of almost all the test suite passing assuming those tests do. This works just like election surveys do

given a small sample, we can predict fairly well that so-and-so percent of the public will vote for candidate A. In our case, the "candidates" are pass or fail, and we can predict how much of the test suite will pass given that sample. Assuming the sample tests all pass, we can say with high likelihood that most of the test suite will in fact pass. (Of course, this is no guarantee, and even a single test failure is serious, however, this gives a quick estimate that your patch does not cause significant and obvious breakage.)

Important Tests

Please see the bottom the file test/test_core.py <https://github.com/emscripten-core/emscripten/blob/main/test/test_core.py>_ for the current test modes, as they may change slowly over time. When you want to run the entire test suite locally, these are currently the important commands:

.. code-block:: bash

Run all core tests

test/runner core*

Run "other" test suite

test/runner other

Run "browser" test suite - this requires a web browser

test/runner browser

Run "sockets" test suite - this requires a web browser too

test/runner sockets

Run "sanity" test suite - this tests setting up emscripten during

first run, etc., and so it modifies your .emscripten file temporarily.

test/runner sanity

Optionally, also run benchmarks to check for regressions

test/runner benchmark

.. _benchmarking:

Benchmarking

Emscripten has a benchmark suite that measures both speed and code size, which includes several interesting real-world codebases, from physics engines to compression libraries to virtual machines. It also includes some existing benchmarks such as CoreMark and LINPACK. See for example this post's section on speed <https://kripken.github.io/blog/wasm/2020/07/27/wasmboxc.html>_ which gives an overview.

To run the benchmark suite, do:

.. code-block:: bash

Run all benchmarks

test/runner benchmark

As with all the test suites, you can also run a specific benchmark:

.. code-block:: bash

Run one specific benchmark

test/runner benchmark.test_skinning

You can also specify which benchmarkers are run by using the environment variable EMTEST_BENCHMARKERS. It accepts a comma separated list of named benchmarkers (names can be found in named_benchmarkers in test/test_benchmark.py).

.. code-block:: bash

Run one specific benchmark and with clang and v8.

EMTEST_BENCHMARKERS=clang,v8 test/runner benchmark.test_skinning

To further customize how the benchmarks are run, you will want to edit the file test/test_benchmark.py. Some of the options include:

DEFAULT_ARG is how long the benchmark should run (they all try to run for a similar amount of time for consistency).
TEST_REPS is how many times to repeat each run (more will take longer, but should have less noise).
PROFILING controls whether the builds are set up for profiling (which can increase code size, so it's not done by default).

Debugging test failures

By default the test runner will hide all test output and use a single terminal line to show test results.

If you want to see more information you can run with -v/--verbose to show the stdout / stderr / logging produced during each test.

If you add a second -v/--verbose then then test runner will output even more information. For example it will show all the subcommands that it runs, as it runs them.

For even more debugging info, you can set :ref:debugging-EMCC_DEBUG which will cause the emscripten compiler itself to output a lot of debug info. This will also tell the compiler to leave all its temporary files behind after it runs (the files go in /tmp/emscripten_temp/):

.. code-block:: bash

On Windows, use "set" to set and un-set the EMCC_DEBUG environment variable:

set EMCC_DEBUG=1 test/runner test_hello_world set EMCC_DEBUG=0

On Linux, you can do this all in one line

EMCC_DEBUG=1 test/runner test_hello_world

EMCC_DEBUG=2 generates additional debug information.

EMCC_DEBUG=2 test/runner test_hello_world

If you run just one a single test (or one test at at time using -j1) the test output can always be found in out/test. This is useful for inspecting test outputs as well as temporary files generated by the test. By default, the temporary directory will be cleaned between each test run, but you can add --no-clean to avoid this.

The :ref:Debugging topic provides more guidance on how to debug Emscripten-generated code.