docs/infra/os_upgrade_playbook.md
[TOC]
Upgrading the operating system (OS) of a Chromium builder or an entire machine fleet (e.g., Ubuntu 20.04 -> 24.04, macOS 14 -> macOS 15, Windows 10 -> Windows 11) is a critical infrastructure operation. It ensures the build infrastructure remains secure, performant, and supported by upstream vendors.
To prevent production outages, tree closures, or Commit Queue (CQ) regressions, OS upgrades follow a phased rollout strategy.
Before initiating an OS upgrade:
Infra>Client>Chrome to track the overall OS upgrade lifecycle.glossary.md for definitions of LUCI,
Swarming, Milo, Starlark, CQ, and CI. For general builder setup, refer to
new_builder.md.go/ineedhw (Google internal).Before migrating a production CQ (try) or CI builder to a new OS version, engineers first create an FYI builder.
This provides the team with a safe environment to validate compile integrity, identify test flakiness or regressions on the new OS, fix missing dependencies, and adjust test parameters prior to fleet rollout.
Locate the Builder Group Starlark File
FYI builder definitions live under
//infra/config/subprojects/chromium/ci/.
Choose or create the relevant OS-specific FYI Starlark file (e.g.,
chromium.fyi.star, chromium.linux.star, or chromium.mac.fyi.star).
Define the FYI Builder in Starlark
For general guidelines on creating builders, see
new_builder.md. Copy the existing production builder
declaration and adjust the following properties:
name: Choose a descriptive name reflecting the target OS version
(e.g., "linux-rel-2404-fyi" or "mac-15-rel-fyi").builder_group: Set to an FYI group such as "chromium.fyi",
"chromium.mac.fyi", or "chromium.win.fyi".os: Specify the new OS swarming dimension using the Starlark helper
(e.g., os = os.LINUX_UBUNTU_24_04 or os = os.MAC_15).tree_closing_notifiers and
gardener_rotations (or set them to None).# Example: FYI builder for testing Ubuntu 24.04 OS upgrade
ci.builder(
name = "linux-rel-2404-fyi",
builder_group = "chromium.fyi",
os = os.LINUX_UBUNTU_24_04,
cores = 32,
# Match GN args and target specs of the production "linux-rel" builder
gn_args = gn_args.config(
# ... matching config ...
),
targets = targets.bundle(
# ... matching targets ...
),
# Ensure no sheriff alerts or tree closing notifications
gardener_rotations = None,
tree_closing_notifiers = None,
)
Commit the Starlark Config Changes Submit a CL with the new FYI builder declaration and verify the builder appears on Milo to validate build and test health on the target OS.
Once the FYI builder is running, use it to gather performance metrics and test health signals before proceeding to fleet-wide migration.
go/ineedhw - Google internal)If build or test execution on the new OS version is slower and the performance regression is unavoidable (e.g., due to updated OS security controls, kernel changes, or toolchain overhead), you must allocate additional hardware capacity to preserve CQ cycle times and CI throughput.
go/estimating-bot-capacity
(Google internal) to determine the target fleet size.
go/ineedhw (Google internal). State the
business justification clearly (e.g., "OS migration for Chromium builder
<builder_name> from OS N to OS N+1 requires 20% additional bot capacity due
to observed runtime regressions").When running the FYI builder on the new OS version, it is common for certain test suites or individual tests to fail due to OS API changes, updated system dependencies, font differences, or updated security policies.
Before migrating production CI/CQ builders to the new OS version, the FYI builder must reach a "green" (passing) state.
If tests or test suites fail on the new OS version, follow this decision process:
┌──────────────────────────────┐
│ Test Fails on New OS Version│
└──────────────┬───────────────┘
│
Is the fix obvious or easy?
│
┌─────────────┴─────────────┐
│ │
[ YES ] [ NO ]
│ │
┌──────────┴──────────┐ ┌──────────┴──────────┐
│ Implement & Submit │ │ Disable Test ONLY │
│ Code/Fix directly │ │ on New OS Version │
└─────────────────────┘ └──────────┴──────────┘
│
┌──────────┴──────────┐
│ File Tracking Bug │
│ against Test Owner │
└─────────────────────┘
If the root cause is straightforward (e.g., a missing OS package dependency, minor flag adjustment, or updated baseline path), the migrating engineer should land a fix directly in the codebase.
If the failure is complex, non-obvious, or outside the migrating engineer's area of expertise, do not let it block the OS upgrade. Instead, selectively disable the failing test or test suite only on the new OS version:
Individual Tests (e.g., Web Tests / Browser Tests / GTest Expectation Files):
Use OS-specific expectation tags in the relevant expectation or filter file
(e.g. TestExpectations, gpu_test_expectations.txt, or filter files) so the
test continues running and passing on older OS versions while being skipped on
the new OS version.
Example (Web Tests / GPU Expectations):
# crbug.com/1234567 [ Ubuntu-24.04 ] fast/canvas/draw-image.html [ Failure ]
# crbug.com/1234568 [ Win11 ] GlTests.SimpleRendering/0 [ Skip ]
Entire Test Suites in Starlark Target Specifications:
If an entire binary test suite fails on the new OS version and cannot run,
temporarily exclude or filter the test suite for the new OS mixin/variant in
//infra/config/targets/:
# Example: Exclude a failing suite specifically on the new OS variant/mixin
targets.matrix_compound_suite(
name = "linux_chromeos_gtests",
matrix = {
"browser_tests": {
"mixins": [
# ...
],
"remove_mixins": [
"ubuntu_24_04", # Skip on Ubuntu 24.04 pending crbug.com/1234569
],
},
},
)
For every test or test suite disabled on the new OS version:
FooBrowserTest.Bar
fails on Ubuntu 24.04 / macOS 15 / Win 11").crbug.com/XXXXXX).Once tests run cleanly ("green") on the new OS version and capacity adjustments are planned, you can begin transitioning the target production builder and machine fleet to the new OS version.
Rather than performing a hard cutover from the old OS version to the new OS version overnight, the standard strategy is to update the builder's accepted swarming dimensions in Starlark to temporarily accept both OS versions.
"OS_Old|OS_New").In Chromium's infrastructure configuration, builders specify abstract OS
constants (e.g., os = os.MAC_DEFAULT), while the central LUCI builder
configuration in main.star and target mixin definitions map these constants to
Swarming dimension OR expressions ("Mac-14|Mac-15").
//infra/config/subprojects/...)Builders reference abstract OS constants rather than hardcoded OS versions:
# Example: Builder referencing os.MAC_DEFAULT
ci.builder(
name = "Mac Builder Siso FYI",
gn_args = "ci/Mac Builder",
os = os.MAC_DEFAULT,
cpu = cpu.ARM64,
)
//infra/config/main.star)During an OS upgrade (e.g., macOS 14 to macOS 15), update
os_dimension_overrides in main.star to map os.MAC_DEFAULT to a dual OS
dimension expression using |:
chromium_luci.configure_builders(
enable_alerts_configuration = True,
os_dimension_overrides = {
# Temporarily accept both Mac-14 and Mac-15 bots during OS migration
os.MAC_DEFAULT: "Mac-14|Mac-15",
os.MAC_BETA: "Mac-15",
os.WINDOWS_DEFAULT: os.WINDOWS_10,
},
)
//infra/config/targets/mixins.star)Similarly, update relevant test mixins to accept tasks on either OS version:
targets.mixin(
name = "mac_arm64",
swarming = targets.swarming(
dimensions = {
"cpu": "arm64",
"os": "Mac-14|Mac-15",
},
),
)
In practice, teams rarely have spare capacity to stand up a parallel fleet of new OS bots to replace 100% of an existing builder's demand upfront. Instead, OS upgrades are conducted as an in-place migration.
To avoid capacity bottlenecks, build queuing, or CQ cycle time spikes, bots assigned to the builder (or shared testing pool) are removed and upgraded in small increments of 5% or less of the total fleet size at any given time.
┌────────────────────────────────────────────────────────────────────────┐
│ Fleet Bot Pool (100% Total) │
└────────────────────────────────────────────────────────────────────────┘
│ │
├── 95%+ Active on Current OS (Serving CQ/CI Workloads) │
│ │
└── <= 5% Quarantined / Re-imaged to New OS (Incremental In-Place) │
For bare-metal physical hosts and connected test devices (such as Mac mini pools, iOS test beds, or Android device labs):
os_dimension_overrides to "Mac-14|Mac-15" (as seen in real-world CLs like
chrome-internal-review CL 9313101
For Google Compute Engine (GCE) cloud-based virtual bot pools, bot allocation
and auto-scaling limits are managed via internal infrastructure configuration
files in infradata/config (Google internal).
Locate GCE Pool Configuration: Open the relevant bot pool definition file
(such as
data/config/configs/chromium-swarm/starlark/bots/chromium/tests.star in
infradata/config).
Real-World Config Example (e.g., Windows 10 -> Windows 11 Migration):
The bot pool (e.g., chromium.tests) defines separate GCE bot declarations
for each OS version (such as chrome.gce_win10_22h2 and
chrome.gce_win11_24h2).
# Win10 test fleet
chrome.gce_win10_22h2(
prefix = "chrome-windows-10-main",
amount = chrome.robocrop(
min_amount = 100,
max_amount = 3700,
),
resource_group = "chrome-desktop-robocrop",
machine_type = "e2-standard-8",
),
# Win11 test fleet
chrome.gce_win11_24h2(
prefix = "chrome-windows-11",
amount = chrome.robocrop(
min_amount = 30,
max_amount = 150,
),
machine_type = "e2-standard-8",
),
Gradually Shift Robocrop Allocation Amounts:
To perform an in-place OS migration for builders using this pool, gradually
decrease the bot capacity (min_amount and max_amount in
chrome.robocrop) assigned to the older OS bot group (Windows 10) by a small
percentage (≤ 5%), and increase the allocation for the new OS bot group
(Windows 11) by the exact same amount.
Land Config & Monitor: Land the infradata/config CL. The GCE provider /
Robocrop autoscaler will automatically drain instances running the old OS
image and spin up replacement instances running the new OS image. Monitor
Swarming queue times and task success rates before performing the next
allocation shift.
Try (Commit Queue) builders pick up their test configurations and compile settings by mirroring one or more CI builders.
Generic CI Builder Migration (No Change Needed):
If the mirrored CI builder itself is migrated to a new OS version (e.g., via
central os_dimension_overrides in main.star), no changes are needed in the
Try builder's declaration.
OS-Versioned CI Builder Migration:
Some Try builders specifically mirror CI builders that explicitly target a
specific OS version.
Example: The android-x64-rel Try/CQ builder specifically mirrors the
ci/android-15-x64-rel CI builder. When upgrading the default CQ builder from
Android 15 to Android 16, the Try builder declaration must be updated to mirror
ci/android-16-x64-rel instead.
In ../../infra/config/subprojects/chromium/try/
(e.g., try.star or try/subproject.star), update the mirrors list and
gn_args config references:
# Example: Updating try builder android-x64-rel to mirror Android 16 CI builder
try_.orchestrator_builder(
name = "android-x64-rel",
mirrors = [
"ci/android-12l-x64-rel-cq",
# Updated from ci/android-15-x64-rel to ci/android-16-x64-rel
"ci/android-16-x64-rel",
"ci/android-webview-13-x64-hostside-rel",
],
gn_args = gn_args.config(
configs = [
# Updated to match new Android 16 CI builder config
"ci/android-16-x64-rel",
"release_try_builder",
],
),
)
When applying OS configuration changes to production CI builders, engineers can temporarily place the builder in experimental mode or disable gardening/sheriff alerts until the migration change is verified to be safe on live infrastructure.
To prevent accidental sheriff alerts or waterfall tree closures while verifying a production builder on a new OS version:
ci.builder(
name = "linux-rel",
# Temporarily set gardener rotations and tree closing to None during OS cutover
gardener_rotations = None,
tree_closing_notifiers = None,
# ...
)
Alternatively, set is_experimental = True (or experimental = True) on the
builder declaration. Experimental builders continue executing builds and
publishing results to Milo, but their build status does not affect CQ or sheriff
rotations.
Once the builder successfully completes several build cycles on the new OS version without infrastructure errors:
gardener_rotations and tree_closing_notifiers.Once 100% of the bot fleet (physical or virtual) has been re-imaged and verified on the new OS version, complete the migration with post-rollout cleanup:
Lock Builder & Mixins to New OS:
Update os_dimension_overrides in main.star and mixin definitions in
mixins.star to point exclusively to the new OS version (e.g., os.MAC_DEFAULT: "Mac-15").
Decommission Temporary FYI Builder:
Remove the temporary FYI builder created in Phase 1 (linux-rel-2404-fyi)
from Starlark.
Clean Up Temporary Test Expectations:
Audit and remove temporary OS expectation tags (e.g., [ Ubuntu-24.04 ] or
[ Win11 ]) or mixin exclusions as test component owners land fixes for
disabled tests.
Land Final Cleanup CL: Land the final Starlark configuration CL to complete the OS upgrade lifecycle.