LoadLine Benchmark

This skill guides the usage of LoadLine benchmarks to measure browser page loading performance by simulating real-world user journeys (loading a web page).

Quick Start

Read third_party/crossbench/config/benchmark/loadline2/README.md for basic info on the LoadLine benchmark.

The entry point for all LoadLine benchmarks is the Crossbench script: third_party/crossbench/cb.py

Common Command Pattern

third_party/crossbench/cb.py <variant> --browser=<browser> --stories=<stories> --repetitions=<N>

Benchmark Variants

Target Browsers (--browser)

• Android (Chrome): Use --browser=adb:chrome (if one device) or --browser=$SERIAL:chrome.
• Local Build: Use the path to the output directory, e.g., --browser=out/Release/chrome.
• System Chrome: Use --browser=chrome-stable, chrome-canary, etc.

Stories (--stories)

Default stories for LoadLine 2:

• amazon_product
• cnn_article
• wikipedia_article
• globo_homepage
• google_search_result

Skip --stories flag to run everything or provide a comma-separated list for specific stories.

Analyzing Results

LoadLine outputs two kinds of numbers: scores and breakdown.

Scores are in runs-per-minute, so higher is better. The main metric is called TOTAL_SCORE, this is the most stable and representative number.

Breakdown values are in milliseconds, so lower is better. Can be useful for analyzing which loading stage was affected in particular.

Statistical Significance

• Noise: Benchmark results are inherently noisy. A single repetition should only be used for smoke-testing.
• Reliability: To confidently determine changes of 1% or higher, run the benchmark with at least 50 repetitions.
• Interpretation: Changes below 1% are typically not considered statistically significant, even with 50 repetitions.

Identifying Tested Version

You can find the exact version of Chrome being tested in the benchmark logs by looking for the 🏷️ STARTING BROWSER Version: line:

🏷️  STARTING BROWSER Version: 138.0.7204.168 stable

Common Workflows

Full Basic Run

Run the full benchmark on a connected Android device:

third_party/crossbench/cb.py loadline2-phone --browser=adb:chrome

Test a feature flag

See if a feature flag affects page loading performance:

third_party/crossbench/cb.py loadline2-phone --browser-config=feature_flag.hjson

where feature_flag.hjson looks like this:

{
  flags: {
    "experiment": {
      "enabled": "--enable-features=YourFeature",
      "disabled": "--disable-features=YourFeature",
    },
  },
  browsers: {
    "chrome": {
      browser: "chrome",
      driver: "adb",
      flags: [ "experiment" ],
    },
  },
}

Compare past versions of Chrome

Check if clank/bin/install_chrome.py exists. If not (chromium-only checkouts), see the next section for building Chromium locally.

Use clank/bin/install_chrome.py to install past versions on the connected device. E.g.

clank/bin/install_chrome.py --channel dev --milestone 146 --signed

will install M146 with "Dev" channel branding. You can use multiple brandings to compare versions on the same device. E.g. install Canary:

clank/bin/install_chrome.py --channel canary --milestone 147 --signed

And then run LoadLine with two browsers:

third_party/crossbench/cb.py loadline2-phone --browser=adb:chrome-dev --browser=adb:chrome-canary

This will compare M146 and M147 on the same device.

Custom-built Chrome

1. Build Chrome for Android with android_channel="canary" gn arg.
2. Uninstall existing Chrome Canary from the device: adb shell pm uninstall com.chrome.canary (this command fails if it's not installed; this is fine and can be ignored)
3. Install the custom built Chrome: adb install out/$OUT_DIR/apks/Chrome.apk
4. Run LoadLine on chrome-canary: third_party/crossbench/cb.py loadline2-phone --browser=adb:chrome-canary

Short run

Before running a full run on a custom-built Chrome, first try with a single repetition

third_party/crossbench/cb.py loadline2-phone --browser=adb:chrome-canary --repetitions=1

Don't trust the numbers, they are going to be very noisy. Do it just to verify that the benchmark runs without errors.