pkg/clusterversion/runbooks/M1_bump_current_version.md
This change advances the current release series version on master after forking a release branch, allowing the addition of new upgrade gates for the next version. It does NOT yet enable mixed-cluster or upgrade tests with the forked release.
When: Around the time the first beta is being cut on the release branch. Technically this can happen right after forking, but if there are changes to gates or upgrades in the forked release it might cause issues with master-to-master upgrades.
Example: After cutting release-25.4, bump master from 25.4 development to 26.1 development.
Before starting, ensure:
release-25.4)master branchThis is the main file where version keys are defined.
Add the new start version constant (around line 238):
// V25_4 is CockroachDB v25.4. It's used for all v25.4.x patch releases.
V25_4
V26_1_Start // Add this line
// *************************************************
// Step (1) Add new versions above this comment.
Add the version to the versionTable (around line 303):
V25_4: {Major: 25, Minor: 4, Internal: 0},
// v26.1 versions. Internal versions must be even.
V26_1_Start: {Major: 25, Minor: 4, Internal: 2}, // Add these lines
// *************************************************
// Step (2): Add new versions above this comment.
Add the placeholder constant (around line 323):
// PreviousRelease is the logical cluster version of the previous release (which must
// have at least an RC build published).
const PreviousRelease Key = V25_3
// V26_1 is a placeholder that will eventually be replaced by the actual 26.1
// version Key, but in the meantime it points to the latest Key. The placeholder
// is defined so that it can be referenced in code that simply wants to check if
// a cluster is running 26.1 and has completed all associated migrations; most
// version gates can use this instead of defining their own version key if they
// only need to check that the cluster has upgraded to 26.1.
const V26_1 = Latest
// DevelopmentBranch must be true on the main development branch but should be
Note: Do NOT update PreviousRelease - that only happens in M.3 after an RC is published.
Add the successor mapping (around line 237):
{25, 2}: {25, 3},
{25, 3}: {25, 4},
{25, 4}: {26, 1}, // Add this line
}
Update the expected release series (around line 96):
expected := "20.1, 20.2, 21.1, 21.2, 22.1, 22.2, 23.1, 23.2, 24.1, 24.2, 24.3, 25.1, 25.2, 25.3, 25.4, 26.1"
Update the bootstrap version (around line 1445):
// Before
var SystemDatabaseSchemaBootstrapVersion = clusterversion.V25_4.Version()
// After
var SystemDatabaseSchemaBootstrapVersion = clusterversion.V26_1_Start.Version()
This ensures new clusters bootstrap at the start of the new version.
Add the first upgrade for the new version (at the end of the upgrades array, around line 124):
upgrade.NewTenantUpgrade(
"create statement_hints table",
clusterversion.V25_4_AddSystemStatementHintsTable.Version(),
upgrade.NoPrecondition,
createStatementHintsTable,
upgrade.RestoreActionNotRequired(
"restore for a cluster predating this table can leave it empty",
),
),
newFirstUpgrade(clusterversion.V26_1_Start.Version()), // Add this line
// Note: when starting a new release version, the first upgrade (for
// Vxy_zStart) must be a newFirstUpgrade. Keep this comment at the bottom.
}
Update the version string:
# Before
v25.4.1
# After
v26.1.0-alpha.00000000
This is the most complex step involving multiple files.
a) Copy current rules to a new release directory:
cp -r pkg/sql/schemachanger/scplan/internal/rules/current \
pkg/sql/schemachanger/scplan/internal/rules/release_25_4
b) Update the package name in all files:
find pkg/sql/schemachanger/scplan/internal/rules/release_25_4 -name "*.go" \
-exec sed -i '' 's/^package current$/package release_25_4/' {} \;
c) Update BUILD.bazel in release_25_4 directory:
Change the library name and import path:
# Before
go_library(
name = "current",
# ...
importpath = "github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/current",
# After
go_library(
name = "release_25_4",
# ...
importpath = "github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_4",
Change the test target name:
# Before
go_test(
name = "current_test",
# ...
embed = [":current"],
# After
go_test(
name = "release_25_4_test",
# ...
embed = [":release_25_4"],
d) Update pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go:
Update the version references:
// Before
const (
// rulesVersion version of elements that can be appended to rel rule names.
rulesVersion = "-25.4"
)
// rulesVersionKey version of elements used by this rule set.
var rulesVersionKey = clusterversion.V25_4
// After
const (
// rulesVersion version of elements that can be appended to rel rule names.
rulesVersion = "-26.1"
)
// rulesVersionKey version of elements used by this rule set.
var rulesVersionKey = clusterversion.V26_1
e) Update pkg/sql/schemachanger/scplan/plan.go:
Add import for the new release:
import (
// ...
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/current"
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_2"
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_3"
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_4" // Add this
// ...
)
Add to rulesForReleases array (around line 158):
var rulesForReleases = []rulesForRelease{
// NB: sort versions in descending order, i.e. newest supported version first.
{activeVersion: clusterversion.Latest, rulesRegistry: current.GetRegistry()},
{activeVersion: clusterversion.V25_4, rulesRegistry: release_25_4.GetRegistry()}, // Add this
{activeVersion: clusterversion.V25_3, rulesRegistry: release_25_3.GetRegistry()},
{activeVersion: clusterversion.V25_2, rulesRegistry: release_25_2.GetRegistry()},
}
a) Update Bazel build files:
./dev gen bazel
This updates various BUILD.bazel files across the codebase.
b) Update releases file:
IMPORTANT: The update-releases-file tool does NOT work for this M.1 task. You must manually update pkg/testutils/release/cockroach_releases.yaml.
Manual Update Steps:
When bumping from version X.Y to X.Z (e.g., 26.1 → 26.2):
"26.1": predecessor: "25.4""26.1":
predecessor: "25.4"
"26.2":
predecessor: "25.4" # Use same predecessor as 26.1
Why: Both 26.1 and 26.2 are development versions that haven't been released yet. They both should upgrade from the last stable release (25.4). This will be corrected later in M.3 when we add actual release data.
Example for 26.1 → 26.2 bump:
# Before (what exists):
"25.4":
latest: 25.4.2
predecessor: "25.3"
"26.1":
predecessor: "25.4"
# After (what you should have):
"25.4":
latest: 25.4.2
predecessor: "25.3"
"26.1":
predecessor: "25.4" # KEEP this entry
"26.2":
predecessor: "25.4" # ADD this entry
# Verify your changes
tail -10 pkg/testutils/release/cockroach_releases.yaml
Note: In M.3, when the RC is published, the predecessor relationships will be updated to reflect the actual release hierarchy.
c) Regenerate scplan test outputs:
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
This updates test data in:
pkg/sql/schemachanger/scplan/internal/rules/current/testdata/deprulespkg/sql/schemachanger/scplan/internal/rules/release_25_4/testdata/deprulesd) Regenerate CLI test outputs:
./dev test pkg/cli -f DeclarativeRules --rewrite
This updates:
pkg/cli/testdata/declarative-rules/invalid_versione) Regenerate documentation and other generated files:
./dev generate
This updates:
docs/generated/settings/settings-for-tenants.txtdocs/generated/settings/settings.htmlRun tests to ensure everything is working:
# Test version packages
./dev test pkg/clusterversion pkg/roachpb
# Test schema changer
./dev test pkg/sql/schemachanger/scplan/internal/rules/...
# Test CLI
./dev test pkg/cli -f DeclarativeRules
A typical M.1 bump should modify approximately 15-20 files:
Core version files:
pkg/clusterversion/cockroach_versions.gopkg/roachpb/version.gopkg/roachpb/version_test.gopkg/sql/catalog/systemschema/system.gopkg/upgrade/upgrades/upgrades.gopkg/build/version.txtGenerated/updated files:
7. pkg/BUILD.bazel
8. pkg/testutils/release/cockroach_releases.yaml
9. pkg/sql/logictest/REPOSITORIES.bzl
10. pkg/cli/testdata/declarative-rules/invalid_version
11. docs/generated/settings/settings-for-tenants.txt
12. docs/generated/settings/settings.html
Schema changer files:
13. pkg/sql/schemachanger/scplan/plan.go
14. pkg/sql/schemachanger/scplan/BUILD.bazel
15. pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go
16. pkg/sql/schemachanger/scplan/internal/rules/current/testdata/deprules
17. pkg/sql/schemachanger/scplan/internal/rules/release_25_4/ (entire new directory)
Cause: The new release directory wasn't created or the package name wasn't updated correctly.
Fix:
ls pkg/sql/schemachanger/scplan/internal/rules/release_25_4grep "^package " pkg/sql/schemachanger/scplan/internal/rules/release_25_4/*.goCause: The placeholder constant wasn't added to cockroach_versions.go.
Fix: Add the const V26_1 = Latest line as shown in step 1.
Cause: Test outputs need to be regenerated after changing version constants.
Fix: Run the rewrite commands:
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
./dev test pkg/cli -f DeclarativeRules --rewrite
Cause: BUILD.bazel files weren't regenerated or are out of sync.
Fix:
./dev gen bazel
Cause: The releases file wasn't updated or cached data is stale.
Fix:
bazel cleanbazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
Cause: Generated documentation files weren't updated.
Fix:
./dev generate
PreviousRelease - This doesn't happen until M.3pkg/testutils/release/cockroach_releases.yaml - Add new version entry, keep previous entries (see step 8b for details)25.4-2, not 26.1-2Before committing, verify:
./dev test pkg/clusterversion pkg/roachpb pkg/sql/schemachanger/scplan/internal/rules/... pkg/cli -f DeclarativeRulesIn the release cycle:
After the main M.1 PR is merged, CI tests will typically reveal failures that need fixing. CRITICAL: Before modifying any code, use the decision tree below to determine the correct approach.
When a test fails after M.1 changes, follow this decision tree:
Test fails after M.1 changes
│
├─ FIRST: Did you modify production code beyond the runbook steps?
│ └─ YES → STOP! Revert changes and consult previous M.1 PRs
│ └─ NO → Continue below
│
├─ Is it a logic test expecting old version number?
│ └─ YES → Update test expectations in testdata file (Type 2A, 2B, 2C below)
│
├─ Is it a bootstrap hash mismatch?
│ └─ YES → Regenerate with -rewrite flag (See Type 2D below)
│
├─ Is it a mixedversion test failure?
│ └─ Did you modify mixedversion production code?
│ ├─ YES → REVERT immediately! Check previous M.1 PRs first
│ └─ NO → Check test history (Type 2E below)
│
└─ Is it failing in production code logic?
└─ Check test history: When was this test added?
├─ BEFORE last M.1 → Likely test expectation issue
└─ AFTER last M.1 → May be new regression test requiring code change (Type 1 below)
RED FLAGS that indicate you're going the wrong direction:
GREEN FLAGS that indicate production code change may be needed:
Before modifying production code, ALWAYS:
gh pr view 149494 --json files --jq '.files[] | select(.path == "path/to/file") | .path'
git log -S "TestName" --oneline --all
git show <commit-hash> # Review the test's purpose
These are logic errors in code that references version gates OR new regression tests added after the previous M.1.
How to identify:
Example locations:
Action:
git log -S "TestSupportsSkipCurrentVersion" --oneline --all
These are straightforward test output updates that happen because version numbers and URLs changed. Do NOT modify production code for these failures.
Pattern: Tests that query version information expect the new version number.
Common files:
pkg/sql/logictest/testdata/logic_test/upgradepkg/sql/logictest/testdata/logic_test/crdb_internalpkg/ccl/logictestccl/testdata/logic_test/crdb_internal_tenantChanges needed:
query T
SELECT crdb_internal.release_series(crdb_internal.node_executable_version())
----
-25.4
+26.1
Quick fix:
# Use replace_all=true for files with multiple occurrences
sed -i '' 's/^25\.4$/26.1/' <file>
Pattern: After bumping to version X.Y, the systemDatabaseSchemaVersion must reflect VX_Y_Start.
File: pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog
How to determine new values:
Find VX_Y_Start in pkg/clusterversion/cockroach_versions.go:
V26_1_Start: {Major: 25, Minor: 4, Internal: 2}
Convert to JSON format:
{
"majorVal": 1000025, // Major * 1000000 + MinorSeries
"minorVal": 4, // Minor from VX_Y_Start
"internal": 2 // Internal from VX_Y_Start
}
Example change:
-"systemDatabaseSchemaVersion": {"internal": 14, "majorVal": 1000025, "minorVal": 3}
+"systemDatabaseSchemaVersion": {"internal": 2, "majorVal": 1000025, "minorVal": 4}
Test to verify:
./dev test pkg/sql/logictest --filter='TestReadCommittedLogic/crdb_internal_catalog'
Pattern: Bootstrap hash values change when system schema versions change.
Symptoms:
TestInitialValuesToString - hash mismatchTestValidateSystemSchemaAfterBootStrap - schema validation failuresFiles typically affected:
pkg/sql/catalog/bootstrap/testdata/testdata (hash values)pkg/sql/catalog/systemschema_test/testdata/bootstrap_systempkg/sql/catalog/systemschema_test/testdata/bootstrap_tenantFix: Regenerate bootstrap test data
bazel test //pkg/sql/catalog/bootstrap:bootstrap_test \
--test_arg=-test.v \
--test_arg=-rewrite \
--sandbox_writable_path=$(pwd)/pkg/sql/catalog/bootstrap
Verification:
# Run test again WITHOUT -rewrite to confirm it passes
./dev test pkg/sql/catalog/bootstrap -f='TestInitialValuesToString'
Pattern: Tests show unexpected upgrade paths or version selections.
⚠️ CRITICAL WARNING: These failures are almost NEVER caused by the version bump itself. They're usually caused by incorrect modifications to mixedversion production code.
Common symptoms:
TestTestPlanner/step_stages - extra version in upgrade pathTest_choosePreviousReleases/skip-version_upgrades - wrong version listTestSupportsSkipUpgradeTo - unexpected true/false valueTestSupportsSkipCurrentVersion - unexpected behaviorBEFORE making any changes:
pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.gogh pr view 149494 --json files --jq '.files[] | select(.path | contains("mixedversion/mixedversion.go"))'
When mixedversion code changes ARE needed: Only if ALL of these are true:
git log -S "TestName")If testdata regeneration is needed (rare):
bazel test //pkg/cmd/roachtest/roachtestutil/mixedversion:mixedversion_test \
--test_filter=TestTestPlanner \
--test_arg=-test.v \
--test_arg=-rewrite \
--sandbox_writable_path=$(pwd)/pkg/cmd/roachtest/roachtestutil/mixedversion
Verification:
# ALWAYS run full mixedversion test suite, not just filtered tests
./dev test pkg/cmd/roachtest/roachtestutil/mixedversion
If unsure: Do NOT modify mixedversion code. Instead, ask on Slack (#test-eng channel) or file an issue.
After fixing test failures:
# Run the tests that previously failed
./dev test pkg/sql/logictest --filter='<test_name>'
# CRITICAL: Run FULL test suites for packages with modified production code
# Do NOT rely on filtered tests alone - they can miss related failures
./dev test pkg/cmd/roachtest/roachtestutil/mixedversion # If mixedversion code changed
./dev test pkg/sql/pgwire -f='TestPGTest' # If pgwire tests failed
./dev test pkg/sql/logictest -f='TestLogic' # After logic test changes
# Verify file count is similar to previous M.1 PRs
# Expected: ~60-65 files changed total (original PR + test fixes)
git diff --stat <base_branch>
# Run broader tests to ensure nothing broke
./dev test pkg/sql/logictest --filter='TestLogic/local/cluster_settings'
./dev test pkg/sql/catalog/bootstrap
Why full test suites matter:
-f='TestSupportsSkipCurrentVersion' can pass while related tests failFiles that SHOULD change in M.1:
pkg/clusterversion/cockroach_versions.go (version constants)pkg/sql/logictest/testdata/logic_test/* (version expectations)pkg/sql/catalog/bootstrap/testdata/* (bootstrap data)pkg/sql/catalog/systemschema_test/testdata/* (schema expectations)pkg/sql/schemachanger/scplan/internal/rules/release_X_Y/* (new release rules)pkg/testutils/release/cockroach_releases.yaml (releases file)docs/generated/settings/* (generated docs)Files that should NOT change in M.1 (unless test is new):
pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go (production logic)pkg/clusterversion/clusterversion.go (logic, only constants change)pkg/sql/logictest/REPOSITORIES.bzl (only changes in M.2 after RC)Files that may change if testdata regeneration is needed:
pkg/cmd/roachtest/roachtestutil/mixedversion/testdata/planner/* (if new regression test)pkg/sql/pgwire/testdata/pgtest/* (URL version updates)This consistency helps validate that the changes are complete and correct.
This task is performed every quarter. Before creating the M.1 PR or fixing any test failures, you MUST validate that the changes follow the same pattern as previous quarterly M.1 PRs.
The most recent M.1 PRs (for reference):
Find the PR for the previous quarter:
# Find recent M.1 version bump PRs
gh pr list --search "clusterversion: move to 25" --state merged --limit 10 \
--json number,title,mergedAt,url | \
jq -r '.[] | select(.title | test("move to [0-9]+\\.[0-9]+ version")) |
"\(.number) | \(.title) | \(.mergedAt) | \(.url)"'
BEFORE creating the PR, compare your changes against the previous M.1 PR:
# Get files changed in your current work
git diff --name-only <base_branch> | sort > /tmp/current_files.txt
# Get files changed in previous M.1 PR (example: PR #149494)
gh pr view 149494 --json files --jq '.files[].path' | sort > /tmp/previous_files.txt
# Compare the two lists
echo "=== Files ONLY in current PR (investigate these!) ==="
comm -13 /tmp/previous_files.txt /tmp/current_files.txt
echo "=== Files ONLY in previous PR (you might be missing these!) ==="
comm -23 /tmp/previous_files.txt /tmp/current_files.txt
echo "=== Common files (expected) ==="
comm -12 /tmp/previous_files.txt /tmp/current_files.txt
For every file that appears in your PR but NOT in the previous M.1 PR, you must:
release update-releases-file - Compare with previous PR to see if the tool modified this file beforeExample investigation:
# Check if REPOSITORIES.bzl was modified in previous M.1 PR
gh pr view 149494 --json files --jq '.files[] | select(.path == "pkg/sql/logictest/REPOSITORIES.bzl") | .path'
# If this returns nothing, the file should NOT change in M.1!
# Check what changed in your version
git diff <base_branch> -- pkg/sql/logictest/REPOSITORIES.bzl
For files that appear in BOTH PRs, verify the changes follow the same pattern:
Expected file changes (from runbook and previous PRs):
| File | Expected Change | Verify |
|---|---|---|
pkg/clusterversion/cockroach_versions.go | Add new version constants, update table | ✓ Always changes |
pkg/build/version.txt | Bump to new alpha version | ✓ Always changes |
pkg/roachpb/version.go | Update successor series map | ✓ Always changes |
pkg/sql/catalog/systemschema/system.go | Update bootstrap version | ✓ Always changes |
pkg/upgrade/upgrades/upgrades.go | Add first upgrade | ✓ Always changes |
pkg/sql/schemachanger/scplan/internal/rules/release_X_Y/* | New directory with copied rules | ✓ Always changes |
pkg/sql/schemachanger/scplan/plan.go | Add new release to rules map | ✓ Always changes |
pkg/testutils/release/cockroach_releases.yaml | Update releases (via tool) | ✓ Always changes |
docs/generated/settings/* | Regenerated docs | ✓ Always changes |
pkg/sql/catalog/bootstrap/testdata/* | Updated bootstrap data | ✓ Always changes |
pkg/BUILD.bazel | Updated build rules | ✓ Always changes |
pkg/sql/logictest/REPOSITORIES.bzl | Should NOT change in M.1 | ⚠️ Only changes in M.2 |
Based on comparing with previous M.1 PRs, these files typically should NOT change:
pkg/sql/logictest/REPOSITORIES.bzl
release update-releases-file modified it, revert the changeTest expectation files (before running tests)
Files unrelated to version bumping
Before creating the M.1 PR, verify:
Real example from PR #156225:
REPOSITORIES.bzl was incorrectly modified by release update-releases-fileTime saved: If we had validated against PR #149494 before creating the PR, we would have caught this immediately and avoided fixing test failures from a broken baseline.
# Step 7: Schema changer setup
cp -r pkg/sql/schemachanger/scplan/internal/rules/current \
pkg/sql/schemachanger/scplan/internal/rules/release_25_4
find pkg/sql/schemachanger/scplan/internal/rules/release_25_4 -name "*.go" \
-exec sed -i '' 's/^package current$/package release_25_4/' {} \;
# Step 8: Regeneration
./dev gen bazel
bazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
./dev test pkg/cli -f DeclarativeRules --rewrite
./dev generate
# Verification
./dev test pkg/clusterversion pkg/roachpb
./dev test pkg/sql/schemachanger/scplan/internal/rules/...
./dev test pkg/cli -f DeclarativeRules