MACRObenchmarking in AndroidX

[TOC]

The public documentation for macrobenchmark explains how to use the library. This page focuses on specifics to writing library macrobenchmarks in the AndroidX repo. If you're looking for measuring CPU perf of individual functions, see the guide for MICRObenchmarks here.

Writing the benchmark

Benchmarks are just regular instrumentation tests! Just use the MacrobenchmarkRule provided by the library:

Kotlin {.new-tab}

    @get:Rule
    val benchmarkRule = MacrobenchmarkRule()

    @Test
    fun startup() = benchmarkRule.measureRepeated(
        packageName = "mypackage.myapp",
        metrics = listOf(StartupTimingMetric()),
        startupMode = StartupMode.COLD,
        iterations = 10
    ) { // this = MacrobenchmarkScope
        pressHome()
        val intent = Intent()
        intent.setPackage("mypackage.myapp")
        intent.setAction("mypackage.myapp.myaction")
        startActivityAndWait(intent)
    }

Java {.new-tab}

    @Rule
    public MacrobenchmarkRule mBenchmarkRule = MacrobenchmarkRule()

    @Test
    public void startup() {
        mBenchmarkRule.measureRepeated(
                "mypackage.myapp",
                Collections.singletonList(new StartupTimingMetric()),
                StartupMode.COLD,
                /* iterations = */ 10,
                scope -> {
                    scope.pressHome();
                    Intent intent = Intent();
                    intent.setPackage("mypackage.myapp");
                    intent.setAction("mypackage.myapp.myaction");
                    scope.startActivityAndWait(intent);
                    return Unit.INSTANCE;
                }
        );
    }

Project structure

As in the public documentation, macrobenchmarks in the AndroidX repo are comprised of an app, and a separate macrobenchmark test module. In the AndroidX repository, there are additional requirements:

1. Macrobenchmark test module path in settings.gradle must end with macrobenchmark to run in CI.

2. Macrobenchmark target module path in settings.gradle should end with macrobenchmark-target to follow convention.

3. Macrobenchmark modules must use project dependencies where available (so implementation(project(":activity:activity-ktx")) rather than implementation("androidx.activity:activity-ktx:1.5.0")). This prevents accidentally testing against out-of-date versions, and increases coverage of lower level libraries.

4. Each library group must declare its own in-group macrobenchmark test and app module, with out using these modules for anything else (e.g. samples). We want to be intentional about which changes affect measurements. More than one of either is allowed, which is sometimes necessary to compare different startup behaviors, see e.g. :emoji2:integration-tests:init-<disabled/enabled>-macrobenchmark-target. Note that comparing multiple app variants are not currently supported by CI.

Compose Macrobenchmark Examples:

• :compose:integration-tests:macrobenchmark-target

• :compose:integration-tests:macrobenchmark

Note: Compose macrobenchmarks are ideally duplicated with View system counterparts, defined in :benchmark:integration-tests:macrobenchmark-target. This is how we compare performance of the two systems.

Setup checklist