Testing WebUI Pages

[TOC]

Chromium WebUI pages should be tested using Mocha test suites. Mocha is a flexible, lightweight test framework with strong async support. With Mocha, we can write small unit tests for individual features of components instead of grouping everything into a monolithic browser test.

There are a few pieces of Chromium WebUI infrastructure designed to make setting up WebUI Mocha browser tests easier:

build_webui_tests is a gn build rule that preprocesses test .ts files, compiles them to JavaScript, and places them in a .grdp file. This file can then be included in the WebUI test .grd so that the files can be served at runtime. This rule is further documented at WebUI Build Configuration
WebUIMochaBrowserTest is a test base class that WebUI tests should inherit from. It handles browser test setup and teardown, injects Mocha, loads the chrome:// page being tested, and imports the test files. This class also ensures code coverage metrics are collected when JS code coverage is enabled. The API for this class is documented in its header file

Note: For the default test loading logic in WebUIMochaBrowserTest to work, test_loader.html, test_loader_util.js and test_loader.js should be registered with the production Web UI's data source. This can be done manually, or (most commonly) can be done by calling SetupWebUIDataSource(). If using the test loader is not feasible for a given test - for example, test relies on loading a specific URL such that loading chrome://foo-bar/test_loader.html would break the test - see Unusual Test Setups

Adding the BrowserTest

UIs should add a .cc file with the definition of their BrowserTest class in chrome/test/data/webui/my_webui_dir_name. The directory structure of chrome/test/data/webui should mirror that of chrome/browser/resources/ where the WebUI TypeScript/HTML code being tested resides. The BrowserTest class defined in the .cc file should inherit from WebUIMochaBrowserTest. Example:

c++

// chrome/test/data/webui/foo_bar/foo_bar_browsertest.cc

#include "base/test/scoped_feature_list.h"
#include "chrome/common/webui_url_constants.h"
#include "chrome/test/base/web_ui_mocha_browser_test.h"
#include "content/public/test/browser_test.h"

class FooBarBrowserTest : public WebUIMochaBrowserTest {
 protected:
  FooBarBrowserTest() {
    set_test_loader_host(chrome::kChromeUIFooBarHost);
  }

 private:
  // Necessary if you have features (or the UI itself) behind a feature flag.
  base::test::ScopedFeatureList scoped_feature_list_{foo_bar::kSomeNewFeature};
};

Note: If testing a chrome-untrusted:// UI, also call:

c++

set_test_loader_scheme(content::kChromeUIUntrustedScheme);

in the constructor.

After defining the class, create one or more IN_PROC_BROWSER_TEST_F functions. Each should run one or more Mocha tests.

c++

typedef FooBarBrowserTest FooBarAppTest;
IN_PROC_BROWSER_TEST_F(FooBarAppTest, All) {
  // Run all Mocha test suites found in app_test
  RunTest("foo_bar/app_test.js", "mocha.run();");
}

Should you write a large number of tests in a single file (e.g., for one particular custom element), it may be necessary to run different suites or even individual tests from that same file in separate IN_PROC_BROWSER_TEST_F invocations, to avoid having one very long test that is at risk of timing out.

Example:

c++

// This class defines a convenience method to run one Mocha test suite in the
// search_box_test.js file at a time.
class FooBarSearchBoxTest: public FooBarBrowserTest {
 protected:
  void RunTestSuite(const std::string& suiteName) {
    FooBarBrowserTest::RunTest(
        "foo_bar/search_box_test.js",
        base::StringPrintf("runMochaSuite('%s');", suiteName.c_str()));
  }
};

IN_PROC_BROWSER_TEST_F(FooBarSearchBoxTest, SearchNonEmpty) {
  RunTestSuite("SearchNonEmpty");
}

IN_PROC_BROWSER_TEST_F(FooBarSearchBoxTest, SearchEmpty) {
  RunTestCase("SearchEmpty");
}

// This class defines a convenience method to run one Mocha test from the
// FooBarChildDialogTest suite at a time.
class FooBarChildDialogTest: public FooBarBrowserTest
 protected:
  void RunTestCase(const std::string& testCase) {
    FooBarBrowserTest::RunTest(
        "foo_bar/child_dialog_test.js",
        base::StringPrintf("runMochaTest('FooBarChildDialogTest', '%s');",
                           testCase.c_str()));
  }
};

IN_PROC_BROWSER_TEST_F(FooBarChildDialogTest, OpenDialog) {
  RunTestCase("OpenDialog");
}

IN_PROC_BROWSER_TEST_F(FooBarChildDialogTest, ClickButtons) {
  RunTestCase("ClickButtons");
}

Adding the Mocha test files

Mocha test files should be added alongside the .cc file in the same directory. Each test file should contain one or more Mocha test suites. Often each major custom element of a UI will have its own Mocha test file, as will many of the data classes.

A build time check exists as part of build_webui_tests() to ensure Mocha test files are actually hooked up to a C++ test file. Mocha test files must end with the \_test.ts suffix. Any other TS supporting files that don't actually hold Mocha test cases should not finish with this suffix otherwise the automated check will trigger erroneously.

WebUIMochaBrowserTest sets up a WebUIDataSource to serve all the test files from chrome://webui-test (or chrome-untrusted://webui-test, if configured via set_test_loader_scheme).

// chrome/test/data/webui/foo_bar/search_box_test.ts

import 'chrome://foo-bar/foo_bar.js';

import type {FooBarSearchBoxElement} from 'chrome://foo-bar/foo_bar.js';
import {assertEquals, assertFalse, assertTrue} from 'chrome://webui-test/chai_assert.js';
import {eventToPromise} from 'chrome://webui-test/test_util.js';

suite('SearchNonEmpty', function() {
  let searchBox: FooBarSearchBoxElement;

  setup(function() {
    document.body.innerHTML = window.trustedTypes!.emptyHTML;
    searchBox = document.createElement('foo-bar-search-box');
    document.body.appendChild(searchBox);

    // More setup goes here
  });

  test('search abc', async function () {
    const search = searchBox.shadowRoot!.querySelector('input');
    assertTrue(!!search);
    search.value = 'abc';
    await eventToPromise('search-change', searchBox);

    // Do some assertions
  });

  test('search def', async function () => {
    // etc
  });
});

suite('SearchEmpty', function() {
  let searchBox: FooBarSearchBoxElement;

  setup(function() {
    document.body.innerHTML = window.trustedTypes!.emptyHTML;
    const searchBox = document.createElement('foo-bar-search-box');
    document.body.appendChild(searchBox);

    // Different setup goes here
  });

  test('search abc when empty', async() {
    const search = searchBox.shadowRoot!.querySelector('input');
    assertTrue(!!search);
    search.value = 'abc';
    await eventToPromise('search-change', searchBox);

    // Do some assertions
  });
});

Setting up the build

To build your tests as part of the browser_tests target, add a BUILD.gn file in your folder:

python

# chrome/test/data/webui/foo_bar/BUILD.gn

import("../build_webui_tests.gni")

build_webui_tests("build") {
  cc_test_files = [ "foo_bar_browsertest.cc" ]

  # Test files go here
  files = [
    "app_test.ts",
    "child_dialog_test.ts",
    "search_box_test.ts",
  ]

  # Path mapping the WebUI host URL to the path where the WebUI's compiled
  # TypeScript files are placed by the TS compiler.
  ts_path_mappings =
      [ "chrome://foo-bar/*|" +
        rebase_path("$root_gen_dir/chrome/browser/resources/foo_bar/tsc/*",
                    target_gen_dir) ]
  ts_deps = [
    "//chrome/browser/resources/foo_bar:build_ts",
  ]
}

Note: If testing a chrome-untrusted:// UI, also set the is_chrome_untrusted parameter for build_webui_tests() to true, to allow importing shared test files from chrome-untrusted://webui-test/ paths.

You then need to hook up the targets generated by this file, and list your browsertest.cc source file, in chrome/test/data/webui/BUILD.gn:

python

# chrome/test/data/webui/BUILD.gn

source_set("browser_tests") {
  testonly = true

  sources = [
    # Lots of files
    # Big list, add your .cc file in alphabetical order
    "foo_bar/foo_bar_browsertest.cc",
    # Lots more files
  ]

  # Conditionally added sources (e.g., platform-specific)

}

# Further down in the file...

generate_grd("build_grd") {
  testonly = true
  grd_prefix = "webui_test"
  output_files_base_dir = "test/data/grit"
  out_grd = "$target_gen_dir/resources.grd"

  deps = [
    ":build_chai_grdp",
    ":build_mocha_grdp",
    # Lots of other deps here
    # Add your build_grdp target:
    "foo_bar:build_grdp",
    # Lots more other deps
  ]

  grdp_files = [
    # Big list of grdp files
    # Add your grdp in alphabetical order
    "$target_gen_dir/foo_bar/resources.grdp",
    # Lots more grdp files
  ]

  # Conditionally added (e.g. platform-specific) grdps and deps go here
}

Unusual Test Setups

Some UIs may need to rely on loading a particular URL, such that using the default chrome://foo-bar/test_loader.html URL that is used to dynamically import the test modules may not work. Other UIs need to do additional setup on the C++ side of the test, and simply want to run a test on a specified WebContents instance. These cases are accommodated by additional methods in WebUIMochaBrowserTest. A few brief examples are provided below.

Example: loading a different URL

c++

IN_PROC_BROWSER_TEST_F(FooBarBrowserTest, NonExistentUrl) {
  // Invoking the test from a non existent URL chrome://foo-bar/does-not-exist/.
  set_test_loader_host(
      std::string(chrome::kChromeFooBarHost) + "/does-not-exist");
  RunTestWithoutTestLoader("foo_bar/non_existent_url_test.js", "mocha.run()");
}

Example: Advanced setup required in C++

c++

IN_PROC_BROWSER_TEST_F(FooDialogBrowserTest, MyTest) {
  ui_test_utils::NavigateToURLWithDisposition(
      browser(), GURL(chrome::kSomeOtherBaseUrl),
      WindowOpenDisposition::CURRENT_TAB,
      ui_test_utils::BROWSER_TEST_WAIT_FOR_LOAD_STOP);
  FooDialog* my_dialog = FooDialog::ShowConstrained(
      kDummyParam,
      browser()->tab_strip_model()->GetActiveWebContents(),
      browser()->window()->GetNativeWindow());
  content::WebContents* web_contents = my_dialog->webui_->GetWebContents();

  ASSERT_TRUE(RunTestOnWebContents(
      web_contents, "foo_dialog/foo_dialog_test.js", "mocha.run()",
      /*skip_test_loader=*/true));
}

Running tests

You can build the browser_tests target and run your WebUI tests just like other browser tests. You may filter tests by class or test name (the arguments to IN_PROC_BROWSER_TEST_F) via --gtest_filter:

autoninja -C out/Release browser_tests
./out/Release/browser_tests --gtest_filter=FooBarSearchBoxTest.Empty

Debugging tests

You can run a subset of the Mocha test cases, by using test.only() instead of test(). For example:

test.only('MyTestcase', function() {...});

You can skip a subset of the Mocha test cases, by using test.skip() instead of test(). For example:

test.skip('MyTestcase', function() {...});

When authoring a new test case or when investigating existing test failures, you will often have a need to step through the code in the DevTools debugger, to figure out what is happening. Use the launchDebugger helper function to do that. Detailed instructions on how to use it exist in the source code documentation here.

Disabling tests

Tests can be disabled at the GTest (C++) level or the Mocha (TypeScript) level. Disabling at the Mocha level is preferred as it allows for more granular control (skipping individual test cases rather than entire suites) and provides better visibility into which specific tests are failing. If the entire suite is flaky or failing, then disabling the GTest might be appropriate.

If your team receives a flaky test bug and you notice that the whole C++ GTest suite is disabled, before deflaking you should first land a CL to disable the specific test cases that are causing problems. After that lands, you can investigate the flakes or failures. This will reduce the chance that other tests in the suite start failing while they are disabled.

Determining which test cases are failing

Test cases can fail or entire suites can time out if they take too long to run. You can expand the "Summary" that appears below the "Failure Reason" and search for the cases marked "failed" or the last test case marked "started" if there is a timeout. Here is an example where the "RenamesBookmark" case failed.

[ RUN      ] SidePanelBookmarksAppTest.General2
[25749:440407:0512/171449.559368:INFO:components/enterprise/browser/controller/chrome_browser_cloud_management_controller.cc:239] No machine level policy manager exists.
Trying to load the allocator multiple times. This is *not* supported.
[25749:440407:0512/171449.854681:INFO:CONSOLE:35] "start", source: chrome://webui-test/mocha_adapter_simple.js (35)
...
[25749:440407:0512/171450.076260:INFO:CONSOLE:44] "    started: General Part2 SetsExpandedSearchResultDescription", source: chrome://webui-test/mocha_adapter_simple.js (44)
[25749:440407:0512/171450.101585:INFO:CONSOLE:47] "     passed: General Part2 SetsExpandedSearchResultDescription", source: chrome://webui-test/mocha_adapter_simple.js (47)
[25749:440407:0512/171450.102566:INFO:CONSOLE:44] "    started: General Part2 RenamesBookmark", source: chrome://webui-test/mocha_adapter_simple.js (44)
[25749:440407:0512/171450.114796:INFO:CONSOLE:58] "     failed: General Part2 RenamesBookmark
AssertionError: expected false to be true
    at assertTrue (chrome://webui-test/chai_assert.js:14:12)
    at Context.<anonymous> (chrome://webui-test/side_panel/bookmarks/power_bookmarks_app_test.js:587:13)", source: chrome://webui-test/mocha_adapter_simple.js (58)
[25749:440407:0512/171450.115013:INFO:CONSOLE:44] "    started: General Part2 BlursRenameInput", source: chrome://webui-test/mocha_adapter_simple.js (44)
[25749:440407:0512/171450.132939:INFO:CONSOLE:47] "     passed: General Part2 BlursRenameInput", source: chrome://webui-test/mocha_adapter_simple.js (47)
...
[25749:440407:0512/171450.385904:INFO:CONSOLE:62] "end: 18/19 ok", source: chrome://webui-test/mocha_adapter_simple.js (62)
[25749:440407:0512/171450.385944:INFO:CONSOLE:64] "TEST all complete, status=FAIL, duration=531ms", source: chrome://webui-test/mocha_adapter_simple.js (64)

Individual test cases also appear as an additional test result. See the below section on how test cases show up in LUCI.

If there is a timeout, it is helpful to look at a few of the failures to see if the same test case is always timing out or different tests (especially ones near the end) are flaking. If different cases are flaking each run, the entire suite will likely need to be disabled while an owner of the tests splits it into two test suites. If the same case is always timing out, then that can be disabled.

Disabling in Mocha (Preferred)

To disable a single test case for all platforms, use test.skip() instead of test():

// TODO(crbug.com/511960512): Flaky test.
test.skip('MyTestcase', function() {...});

To disable an entire suite, use suite.skip() instead of suite():

// TODO(crbug.com/511960512): Flaky test.
suite.skip('MySuite', function() {...});

To disable a test for a specific platform, use <if expr>:

// TODO(crbug.com/511960512): Flaky test.
// <if expr="not is_macosx">
test('LogsBookmarkCountMetric', async () => {
  ...
}
// </if>

Disabling in GTest (Discouraged)

Disabling a C++ GTest suite using the DISABLED_ prefix is discouraged. There are often dozens of individual test cases under a single C++ GTest suite. Disabling the whole suite when a single test is failing results in decreased test coverage and a higher change that regressions could be introduced.

Here is how to disable a test in GTest if that is required:

c++

// TODO(crbug.com/511960512): Flaky test.
IN_PROC_BROWSER_TEST_F(SidePanelBookmarksAppTest, DISABLED_General2) {
  SidePanelBookmarksTest::RunTest(
      "side_panel/bookmarks/power_bookmarks_app_test.js",
      "runMochaSuite('General Part2');");
}

A presubmit check enforced in chrome/test/data/webui/ will trigger if DISABLED_ is added to C++ test files. This can be bypassed by adding SKIP_DISABLING_SUITES_CHECK: <reason> to the CL description if disabling at the C++ level is truly necessary (e.g., the entire test suite is flaky or failing).

Reporting WebUI test results to LUCI

WebUI test results are automatically reported to LUCI. No special handling is required of WebUI test authors.

A single GTest typically reports a single test result. A new test reporting mechanism was added to support reporting multiple results from a single GTest. Using this mechanism, each Mocha JS test can have its own entry in LUCI (see screenshot). The integration is accomplished by calling base::AddSubTestResult in WebUIMochaBrowserTest.

Common errors

Tests that rely on focus, blur, or other input-related events need to be added to the interactive_ui_tests build target rather than the browser_tests target. browser_tests are run in parallel, and the window running the test will gain/lose activation at unpredictable times.

Tests that touch production logic that depends on onFocus, onBlur or similar events also needs to be added to interactive_ui_tests