Bisecting Performance Regressions

The perf tests on chromium's continuous build are very long-running, so we cannot run them on every revision. Further, separate repositories like v8 and skia sometimes roll multiple performance-sensitive changes into chromium at once. For these reasons, we need a tool that can bisect the root cause of performance regressions over a CL range, descending into third_party repositories as necessary. The name of the service that does this is called Pinpoint.

[TOC]

Starting a perf bisect

Performance bisects are integrated with the Chrome Performance Dashboard and monorail. Users kick off perf bisects on the perf dashboard and view results in monorail.

You can kick off perf bisect from performance graphs on the perf dashboard for any test that runs on the chromium.perf waterfall.

To get to a graph, use one of the following methods:

• From a perf sheriff-filed bug, follow the link in #1 that looks like https://chromeperf.appspot.com/group_report?bug_id=XXXXXX. Check the boxes next to alerts in the table to display graphs.
• From the alerts page, check the box next to an alert and click the Graph button.
• From the report page, use the menu to navigate to the graph you want.

To kick off a bisect from the graph:

1. Click on a data point in the graph.
2. In the tooltip that shows up, click the BISECT button.
3. Make sure to enter a Bug ID in the dialog that comes up.
4. Click the CREATE button.


What are all the boxes in the form?

• Bug ID: The bug number in monorail. It's very important to fill in this field, as this is where bisect results will be posted.
• Start commit: The chromium commit pos to start bisecting from. This is prefilled by the dashboard to the start of the revision range for the point you clicked on. You can set it to an earlier commit position to bisect a larger range.
• End commit: The chromium commit pos to bisect to. This is prefilled by the dashboard to the end of the revision range for the point you clicked on. You can set it to a later commit pos to bisect a larger range.
• Story Filter: This is a flag specific to telemetry. It tells telemetry to only run a specific test case, instead of running all the test cases in the suite. This dramatically reduces bisect time for large test suites. The dashboard will prefill this box based on the graph you clicked on. If you suspect that test cases in the benchmark are not independent, you can try bisecting with this box cleared.
• Performance or functional: use "performance" to bisect on a performance metric, or "functional" to bisect on a test failure or flake.

Interpreting the results

The bisect bot will output a comment on the bug when the bisection is complete. See Understanding the Bisect Results for details on how to interpret the results.

Traces and stdout

On the Job result page, there is a line chart. Each dot represents a commit. The bisect culprits are represented by flashing dots. Clicking on a dot reveals some colored bars; each box represents one benchmark run. Click on one of the runs to see trace links. Click on the task_id link to see the stdout.