Performance measurements

This directory contains scripts that measure the start-up performance of the Theia frontend in both the browser and the Electron examples.

The frontend's start-up time is measured using the timestamp of the last recorded Largest contentful paint (LCP) candidate metric. In addition, the scripts scrape Theia's Stopwatch log output to capture complementary metrics; see Log-based metrics below.

Running the browser start-up script

Quick Start

Execute npm run performance:startup:browser in the root directory to startup the backend and execute the script.

Prerequisites

To run the script the Theia backend needs to be started. This can either be done with the Launch Browser Backend launch config or by running npm run start in the examples/browser-app directory. For the per-contribution Stopwatch breakdown to be captured, the backend must be started at debug log level (e.g. npm run start -- --log-level=debug); otherwise only aggregate metrics will appear in the script's summary. The bundled npm run performance:startup:browser flow already does this.

Executing the script

The script can be executed using node browser-performance.js in this directory.

The script accepts the following optional parameters:

• --name: Specify a name for the current measurement (default: Browser Frontend Startup)
• --url: Point Theia to a url for example for specifying a specific workspace (default: http://localhost:3000/#/<pathToMeasurementScript>/workspace)
• --folder: Folder name for the generated tracing files in the profiles folder (default: browser)
• --runs: Number of runs for the measurement (default: 10)
• --headless: Boolean, if the tests should be run in headless mode (default: true)

Note: When multiple runs are specified the script will calculate the mean and the standard deviation of all values.

Running the Electron start-up script

Quick Start

Execute yarn run performance:startup:electron in the root directory to execute the script.

Prerequisites

To run the script the Theia Electron example needs to be built. In the root directory:

npm install
npm run build:electron

Executing the script

The script can be executed using node electron-performance.js in this directory.

The script accepts the following optional parameters:

• --name: Specify a name for the current measurement (default: Electron Frontend Startup)
• --folder: Folder name for the generated tracing files in the profiles folder (default: electron)
• --workspace: Absolute path to a Theia workspace to open (default: an empty workspace folder)
• --runs: Number of runs for the measurement (default: 10)
• --debug: Whether to log debug information to the console. Currently, this is only the standard error of the Electron app, which ordinarily is suppressed because the child process is detached

Note: When multiple runs are specified the script will calculate the mean and the standard deviation of all values, except for any runs that failed to capture a measurement due to an exception.

It can happen that the Electron app does not start normally because the native browser modules are not properly built for the Electron target. The symptom for this is usually an error about a module not self-registering; when this condition is detected, the script stops rather than print out an inevitable series of failures to measure the performance.

Measure impact on startup performance of extensions

To measure the startup performance impact that extensions have on the application, another script is available, which uses the measurements from the browser-performance.js or electron-performance.js script. The extension-impact.js script runs the measurement for a defined base application (base-package.json in this directory) and then measures the startup time when one of the defined extensions is added to the base application. The script will then print a table (in CSV format) to the console (and store it in a file) which contains the mean, standard deviation (Std Dev) and coefficient of variation (CV) for each extensions run. Additionally, each extensions entry will contain the difference to the base application time.

Example Table:

Script usage

The script can be executed by running node extension-impact.js in this directory.

The following parameters are available:

• --app: The example app in which to measure performance, either browser or electron (default: browser)

• --runs: Specify the number of measurements for each extension (default: 10)

• --base-time: Provide an existing measurement (mean) for the base Theia application. If none is provided it will be measured.

• --extensions: Provide a list of extensions (need to be locally installed) that shall be tested (default: all extensions in packages folder)

  Note: Each entry should:

  • have the format {name}:{version}

  • not contain whitespaces

  • and be separated by whitespaces

    For example: --extensions @theia/core:1.19.0 @theia/keymaps:1.19.0

• --yarn: Flag to trigger a full build at script startup (e.g. to build changes to extensions)

• --url: Specify a URL that Theia should be launched with (can be used to specify the workspace to be opened). Applies only to the browser app (default: http://localhost:3000/#/<GIT_ROOT>/scripts/performance/workspace)

• --workspace: Specify a workspace on which to launch Theia. Applies only to the electron app (default: /<GIT_ROOT>/scripts/performance/workspace)

• --file: Relative path to the main output file (default: ./script.csv)

• --detail-file: Relative path to a second CSV that records per-contribution breakdowns (default: derived from --file by inserting -details before the .csv suffix)

Note: If no extensions are provided all extensions from the packages folder will be measured.

Per-contribution detail output

Alongside the main LCP comparison CSV, extension-impact.js emits a detail CSV with one row per (extension, metric) pair. Metrics include the All frontend/backend contributions settled aggregate and every per-contribution timing that the Theia Stopwatch emitted during the runs. This lets you attribute an extension's startup cost to the specific contribution(s) responsible.

Example detail table:

Log-based metrics

The Theia Stopwatch emits startup timing logs in a well-defined format that the performance scripts scrape:

<activity>: <ms> ms [<seconds> s since {backend process|frontend page} start]

Log lines the scripts currently capture:

• All frontend contributions settled: logged by the frontend once all contributions whose lifecycle methods returned promises have resolved (i.e. steady state, as opposed to first paint).
• All backend contributions settled: the backend equivalent.
• Frontend <ContributionName>.<phase>: per-contribution timing for initialize, configure, onStart, initializeLayout, and onDidInitializeLayout.
• Backend <ContributionName>.<phase>: per-contribution timing for initialize, configure, and onStart.
• Frontend <ContributionName> settled / Backend <ContributionName> settled: per-contribution settlement, logged only when a contribution returned promises from more than one lifecycle method.

Where these logs are visible:

Theia's frontend logger forwards all console output to the backend over RPC, so the Electron child process's stdout/stderr contains both frontend and backend Stopwatch lines. The browser script cannot observe backend logs because the backend runs as a separate process whose stdout is not captured.