pkg/clusterversion/runbooks/M1_bump_current_version.md
This change advances the current release series version on master after forking a release branch, allowing the addition of new upgrade gates for the next version. It does NOT yet enable mixed-cluster or upgrade tests with the forked release.
When: Around the time the first beta is being cut on the release branch. Technically this can happen right after forking, but if there are changes to gates or upgrades in the forked release it might cause issues with master-to-master upgrades.
Example: After cutting release-25.4, bump master from 25.4 development to 26.1 development.
Before starting, ensure:
release-25.4)master branchThis is the main file where version keys are defined.
Add the new start version constant (around line 238):
// V25_4 is CockroachDB v25.4. It's used for all v25.4.x patch releases.
V25_4
V26_1_Start // Add this line
// *************************************************
// Step (1) Add new versions above this comment.
Add the version to the versionTable (around line 303):
V25_4: {Major: 25, Minor: 4, Internal: 0},
// v26.1 versions. Internal versions must be even.
V26_1_Start: {Major: 25, Minor: 4, Internal: 2}, // Add these lines
// *************************************************
// Step (2): Add new versions above this comment.
Add the placeholder constant (around line 323):
// PreviousRelease is the logical cluster version of the previous release (which must
// have at least an RC build published).
const PreviousRelease Key = V25_3
// V26_1 is a placeholder that will eventually be replaced by the actual 26.1
// version Key, but in the meantime it points to the latest Key. The placeholder
// is defined so that it can be referenced in code that simply wants to check if
// a cluster is running 26.1 and has completed all associated migrations; most
// version gates can use this instead of defining their own version key if they
// only need to check that the cluster has upgraded to 26.1.
const V26_1 = Latest
// DevelopmentBranch must be true on the main development branch but should be
Note: Do NOT update PreviousRelease - that only happens in M.3 after an RC is published.
Add the successor mapping (around line 237):
{25, 2}: {25, 3},
{25, 3}: {25, 4},
{25, 4}: {26, 1}, // Add this line
}
Update the expected release series (around line 96):
expected := "20.1, 20.2, 21.1, 21.2, 22.1, 22.2, 23.1, 23.2, 24.1, 24.2, 24.3, 25.1, 25.2, 25.3, 25.4, 26.1"
Update the bootstrap version (around line 1445):
// Before
var SystemDatabaseSchemaBootstrapVersion = clusterversion.V25_4.Version()
// After
var SystemDatabaseSchemaBootstrapVersion = clusterversion.V26_1_Start.Version()
This ensures new clusters bootstrap at the start of the new version.
Add the first upgrade for the new version (at the end of the upgrades array, around line 124):
upgrade.NewTenantUpgrade(
"create statement_hints table",
clusterversion.V25_4_AddSystemStatementHintsTable.Version(),
upgrade.NoPrecondition,
createStatementHintsTable,
upgrade.RestoreActionNotRequired(
"restore for a cluster predating this table can leave it empty",
),
),
newFirstUpgrade(clusterversion.V26_1_Start.Version()), // Add this line
// Note: when starting a new release version, the first upgrade (for
// Vxy_zStart) must be a newFirstUpgrade. Keep this comment at the bottom.
}
Update the version string:
# Before
v25.4.1
# After
v26.1.0-alpha.00000000
This is the most complex step involving multiple files.
a) Copy current rules to a new release directory:
cp -r pkg/sql/schemachanger/scplan/internal/rules/current \
pkg/sql/schemachanger/scplan/internal/rules/release_25_4
b) Update the package name in all files:
find pkg/sql/schemachanger/scplan/internal/rules/release_25_4 -name "*.go" \
-exec sed -i '' 's/^package current$/package release_25_4/' {} \;
c) Update BUILD.bazel in release_25_4 directory:
Change the library name and import path:
# Before
go_library(
name = "current",
# ...
importpath = "github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/current",
# After
go_library(
name = "release_25_4",
# ...
importpath = "github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_4",
Change the test target name:
# Before
go_test(
name = "current_test",
# ...
embed = [":current"],
# After
go_test(
name = "release_25_4_test",
# ...
embed = [":release_25_4"],
d) Update pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go:
Update the version references:
// Before
const (
// rulesVersion version of elements that can be appended to rel rule names.
rulesVersion = "-25.4"
)
// rulesVersionKey version of elements used by this rule set.
var rulesVersionKey = clusterversion.V25_4
// After
const (
// rulesVersion version of elements that can be appended to rel rule names.
rulesVersion = "-26.1"
)
// rulesVersionKey version of elements used by this rule set.
var rulesVersionKey = clusterversion.V26_1
e) Update pkg/sql/schemachanger/scplan/plan.go:
Add import for the new release:
import (
// ...
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/current"
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_2"
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_3"
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_4" // Add this
// ...
)
Add to rulesForReleases array (around line 158):
var rulesForReleases = []rulesForRelease{
// NB: sort versions in descending order, i.e. newest supported version first.
{activeVersion: clusterversion.Latest, rulesRegistry: current.GetRegistry()},
{activeVersion: clusterversion.V25_4, rulesRegistry: release_25_4.GetRegistry()}, // Add this
{activeVersion: clusterversion.V25_3, rulesRegistry: release_25_3.GetRegistry()},
{activeVersion: clusterversion.V25_2, rulesRegistry: release_25_2.GetRegistry()},
}
a) Update Bazel build files:
./dev gen bazel
This updates various BUILD.bazel files across the codebase.
b) Update releases file:
bazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
This updates pkg/testutils/release/cockroach_releases.yaml. Note that the forked release version will disappear from the releases file. This is expected and desired - if it's there, upgrade tests will attempt to run against it.
IMPORTANT - Predecessor Version Edge Case:
The release update-releases-file command assumes M.1 is performed shortly after the branch cut, before the RC is published. If you're performing M.1 after the RC is already published (e.g., 25.4.0-rc.1 exists when bumping to 26.1), the tool will incorrectly set the predecessor.
Example: When bumping to 26.1 after 25.4.0-rc.1 is published:
"26.1": predecessor: "25.4""26.1": predecessor: "25.3"Why: The new major version (26.1) should upgrade from the previous major series (25.3), not the just-forked series (25.4).
Fix: After running update-releases-file, check the generated entry for the new version and manually correct the predecessor if needed:
# Check what was generated
grep -A 1 '"26.1":' pkg/testutils/release/cockroach_releases.yaml
# If predecessor is "25.4", manually edit to "25.3"
c) Regenerate scplan test outputs:
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
This updates test data in:
pkg/sql/schemachanger/scplan/internal/rules/current/testdata/deprulespkg/sql/schemachanger/scplan/internal/rules/release_25_4/testdata/deprulesd) Regenerate CLI test outputs:
./dev test pkg/cli -f DeclarativeRules --rewrite
This updates:
pkg/cli/testdata/declarative-rules/invalid_versione) Regenerate documentation and other generated files:
./dev generate
This updates:
docs/generated/settings/settings-for-tenants.txtdocs/generated/settings/settings.htmlRun tests to ensure everything is working:
# Test version packages
./dev test pkg/clusterversion pkg/roachpb
# Test schema changer
./dev test pkg/sql/schemachanger/scplan/internal/rules/...
# Test CLI
./dev test pkg/cli -f DeclarativeRules
A typical M.1 bump should modify approximately 15-20 files:
Core version files:
pkg/clusterversion/cockroach_versions.gopkg/roachpb/version.gopkg/roachpb/version_test.gopkg/sql/catalog/systemschema/system.gopkg/upgrade/upgrades/upgrades.gopkg/build/version.txtGenerated/updated files:
7. pkg/BUILD.bazel
8. pkg/testutils/release/cockroach_releases.yaml
9. pkg/sql/logictest/REPOSITORIES.bzl
10. pkg/cli/testdata/declarative-rules/invalid_version
11. docs/generated/settings/settings-for-tenants.txt
12. docs/generated/settings/settings.html
Schema changer files:
13. pkg/sql/schemachanger/scplan/plan.go
14. pkg/sql/schemachanger/scplan/BUILD.bazel
15. pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go
16. pkg/sql/schemachanger/scplan/internal/rules/current/testdata/deprules
17. pkg/sql/schemachanger/scplan/internal/rules/release_25_4/ (entire new directory)
Cause: The new release directory wasn't created or the package name wasn't updated correctly.
Fix:
ls pkg/sql/schemachanger/scplan/internal/rules/release_25_4grep "^package " pkg/sql/schemachanger/scplan/internal/rules/release_25_4/*.goCause: The placeholder constant wasn't added to cockroach_versions.go.
Fix: Add the const V26_1 = Latest line as shown in step 1.
Cause: Test outputs need to be regenerated after changing version constants.
Fix: Run the rewrite commands:
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
./dev test pkg/cli -f DeclarativeRules --rewrite
Cause: BUILD.bazel files weren't regenerated or are out of sync.
Fix:
./dev gen bazel
Cause: The releases file wasn't updated or cached data is stale.
Fix:
bazel cleanbazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
Cause: Generated documentation files weren't updated.
Fix:
./dev generate
PreviousRelease - This doesn't happen until M.3pkg/testutils/release/cockroach_releases.yaml - This doesn't happen until M.325.4-2, not 26.1-2Before committing, verify:
./dev test pkg/clusterversion pkg/roachpb pkg/sql/schemachanger/scplan/internal/rules/... pkg/cli -f DeclarativeRulesIn the release cycle:
After the main M.1 PR is merged, CI tests will typically reveal two types of failures that need fixing:
These are logic errors in code that references version gates. Review test output carefully as each case is unique and requires understanding the specific version gate logic.
Example locations:
These are straightforward test output updates that happen because version numbers and URLs changed.
Pattern: After bumping from X.Y to next major version, all doc URLs referencing vX.Y must change to dev.
Files typically affected:
pkg/sql/logictest/testdata/logic_test/*pkg/sql/pgwire/testdata/pgtest/*pkg/ccl/logictestccl/testdata/logic_test/*Changes needed:
# Documentation URLs
www.cockroachlabs.com/docs/v25.4/* → www.cockroachlabs.com/docs/dev/*
# Issue tracker URLs
go.crdb.dev/issue-v/*/v25.4 → go.crdb.dev/issue-v/*/dev
Quick fix:
# Find all files with v25.4 references
grep -r "v25\.4" pkg/sql pkg/ccl --include="*.md" --include="logic_test*" --include="pgtest*"
# Replace with sed (example for URLs)
sed -i '' 's|www.cockroachlabs.com/docs/v25.4|www.cockroachlabs.com/docs/dev|g' <file>
sed -i '' 's|go.crdb.dev/issue-v/\([0-9]*\)/v25.4|go.crdb.dev/issue-v/\1/dev|g' <file>
Pattern: Tests that query version information expect the new version number.
Common files:
pkg/sql/logictest/testdata/logic_test/upgradepkg/sql/logictest/testdata/logic_test/crdb_internalpkg/ccl/logictestccl/testdata/logic_test/crdb_internal_tenantChanges needed:
query T
SELECT crdb_internal.release_series(crdb_internal.node_executable_version())
----
-25.4
+26.1
Quick fix:
# Use replace_all=true for files with multiple occurrences
sed -i '' 's/^25\.4$/26.1/' <file>
Pattern: After bumping to version X.Y, the systemDatabaseSchemaVersion must reflect VX_Y_Start.
File: pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog
How to determine new values:
Find VX_Y_Start in pkg/clusterversion/cockroach_versions.go:
V26_1_Start: {Major: 25, Minor: 4, Internal: 2}
Convert to JSON format:
{
"majorVal": 1000025, // Major * 1000000 + MinorSeries
"minorVal": 4, // Minor from VX_Y_Start
"internal": 2 // Internal from VX_Y_Start
}
Example change:
-"systemDatabaseSchemaVersion": {"internal": 14, "majorVal": 1000025, "minorVal": 3}
+"systemDatabaseSchemaVersion": {"internal": 2, "majorVal": 1000025, "minorVal": 4}
Test to verify:
./dev test pkg/sql/logictest --filter='TestReadCommittedLogic/crdb_internal_catalog'
After fixing test failures:
# Run the tests that previously failed
./dev test pkg/sql/logictest --filter='<test_name>'
# Verify file count is similar to previous M.1 PRs
# Expected: ~60-65 files changed total (original PR + test fixes)
git diff --stat <base_branch>
# Run broader tests to ensure nothing broke
./dev test pkg/sql/logictest --filter='TestLogic/local/cluster_settings'
./dev test pkg/sql/catalog/bootstrap
This consistency helps validate that the changes are complete and correct.
This task is performed every quarter. Before creating the M.1 PR or fixing any test failures, you MUST validate that the changes follow the same pattern as previous quarterly M.1 PRs.
The most recent M.1 PRs (for reference):
Find the PR for the previous quarter:
# Find recent M.1 version bump PRs
gh pr list --search "clusterversion: move to 25" --state merged --limit 10 \
--json number,title,mergedAt,url | \
jq -r '.[] | select(.title | test("move to [0-9]+\\.[0-9]+ version")) |
"\(.number) | \(.title) | \(.mergedAt) | \(.url)"'
BEFORE creating the PR, compare your changes against the previous M.1 PR:
# Get files changed in your current work
git diff --name-only <base_branch> | sort > /tmp/current_files.txt
# Get files changed in previous M.1 PR (example: PR #149494)
gh pr view 149494 --json files --jq '.files[].path' | sort > /tmp/previous_files.txt
# Compare the two lists
echo "=== Files ONLY in current PR (investigate these!) ==="
comm -13 /tmp/previous_files.txt /tmp/current_files.txt
echo "=== Files ONLY in previous PR (you might be missing these!) ==="
comm -23 /tmp/previous_files.txt /tmp/current_files.txt
echo "=== Common files (expected) ==="
comm -12 /tmp/previous_files.txt /tmp/current_files.txt
For every file that appears in your PR but NOT in the previous M.1 PR, you must:
release update-releases-file - Compare with previous PR to see if the tool modified this file beforeExample investigation:
# Check if REPOSITORIES.bzl was modified in previous M.1 PR
gh pr view 149494 --json files --jq '.files[] | select(.path == "pkg/sql/logictest/REPOSITORIES.bzl") | .path'
# If this returns nothing, the file should NOT change in M.1!
# Check what changed in your version
git diff <base_branch> -- pkg/sql/logictest/REPOSITORIES.bzl
For files that appear in BOTH PRs, verify the changes follow the same pattern:
Expected file changes (from runbook and previous PRs):
| File | Expected Change | Verify |
|---|---|---|
pkg/clusterversion/cockroach_versions.go | Add new version constants, update table | ✓ Always changes |
pkg/build/version.txt | Bump to new alpha version | ✓ Always changes |
pkg/roachpb/version.go | Update successor series map | ✓ Always changes |
pkg/sql/catalog/systemschema/system.go | Update bootstrap version | ✓ Always changes |
pkg/upgrade/upgrades/upgrades.go | Add first upgrade | ✓ Always changes |
pkg/sql/schemachanger/scplan/internal/rules/release_X_Y/* | New directory with copied rules | ✓ Always changes |
pkg/sql/schemachanger/scplan/plan.go | Add new release to rules map | ✓ Always changes |
pkg/testutils/release/cockroach_releases.yaml | Update releases (via tool) | ✓ Always changes |
docs/generated/settings/* | Regenerated docs | ✓ Always changes |
pkg/sql/catalog/bootstrap/testdata/* | Updated bootstrap data | ✓ Always changes |
pkg/BUILD.bazel | Updated build rules | ✓ Always changes |
pkg/sql/logictest/REPOSITORIES.bzl | Should NOT change in M.1 | ⚠️ Only changes in M.2 |
Based on comparing with previous M.1 PRs, these files typically should NOT change:
pkg/sql/logictest/REPOSITORIES.bzl
release update-releases-file modified it, revert the changeTest expectation files (before running tests)
Files unrelated to version bumping
Before creating the M.1 PR, verify:
Real example from PR #156225:
REPOSITORIES.bzl was incorrectly modified by release update-releases-fileTime saved: If we had validated against PR #149494 before creating the PR, we would have caught this immediately and avoided fixing test failures from a broken baseline.
# Step 7: Schema changer setup
cp -r pkg/sql/schemachanger/scplan/internal/rules/current \
pkg/sql/schemachanger/scplan/internal/rules/release_25_4
find pkg/sql/schemachanger/scplan/internal/rules/release_25_4 -name "*.go" \
-exec sed -i '' 's/^package current$/package release_25_4/' {} \;
# Step 8: Regeneration
./dev gen bazel
bazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
./dev test pkg/cli -f DeclarativeRules --rewrite
./dev generate
# Verification
./dev test pkg/clusterversion pkg/roachpb
./dev test pkg/sql/schemachanger/scplan/internal/rules/...
./dev test pkg/cli -f DeclarativeRules