Back to Cockroach

M.1: Bump Current Version (Master Branch)

pkg/clusterversion/runbooks/M1_bump_current_version.md

26.2.029.7 KB
Original Source

M.1: Bump Current Version (Master Branch)

This change advances the current release series version on master after forking a release branch, allowing the addition of new upgrade gates for the next version. It does NOT yet enable mixed-cluster or upgrade tests with the forked release.

When: Around the time the first beta is being cut on the release branch. Technically this can happen right after forking, but if there are changes to gates or upgrades in the forked release it might cause issues with master-to-master upgrades.

Example: After cutting release-25.4, bump master from 25.4 development to 26.1 development.

Prerequisites

Before starting, ensure:

  1. The release branch has been cut (e.g., release-25.4)
  2. You're working on the master branch
  3. You know the previous and new release numbers (e.g., 25.4 → 26.1)

Step-by-Step Checklist

1. Update pkg/clusterversion/cockroach_versions.go

This is the main file where version keys are defined.

Add the new start version constant (around line 238):

go
// V25_4 is CockroachDB v25.4. It's used for all v25.4.x patch releases.
V25_4

V26_1_Start  // Add this line

// *************************************************
// Step (1) Add new versions above this comment.

Add the version to the versionTable (around line 303):

go
V25_4: {Major: 25, Minor: 4, Internal: 0},

// v26.1 versions. Internal versions must be even.
V26_1_Start: {Major: 25, Minor: 4, Internal: 2},  // Add these lines

// *************************************************
// Step (2): Add new versions above this comment.

Add the placeholder constant (around line 323):

go
// PreviousRelease is the logical cluster version of the previous release (which must
// have at least an RC build published).
const PreviousRelease Key = V25_3

// V26_1 is a placeholder that will eventually be replaced by the actual 26.1
// version Key, but in the meantime it points to the latest Key. The placeholder
// is defined so that it can be referenced in code that simply wants to check if
// a cluster is running 26.1 and has completed all associated migrations; most
// version gates can use this instead of defining their own version key if they
// only need to check that the cluster has upgraded to 26.1.
const V26_1 = Latest

// DevelopmentBranch must be true on the main development branch but should be

Note: Do NOT update PreviousRelease - that only happens in M.3 after an RC is published.

2. Update pkg/roachpb/version.go

Add the successor mapping (around line 237):

go
{25, 2}: {25, 3},
{25, 3}: {25, 4},
{25, 4}: {26, 1},  // Add this line
}

3. Update pkg/roachpb/version_test.go

Update the expected release series (around line 96):

go
expected := "20.1, 20.2, 21.1, 21.2, 22.1, 22.2, 23.1, 23.2, 24.1, 24.2, 24.3, 25.1, 25.2, 25.3, 25.4, 26.1"

4. Update pkg/sql/catalog/systemschema/system.go

Update the bootstrap version (around line 1445):

go
// Before
var SystemDatabaseSchemaBootstrapVersion = clusterversion.V25_4.Version()

// After
var SystemDatabaseSchemaBootstrapVersion = clusterversion.V26_1_Start.Version()

This ensures new clusters bootstrap at the start of the new version.

5. Update pkg/upgrade/upgrades/upgrades.go

Add the first upgrade for the new version (at the end of the upgrades array, around line 124):

go
	upgrade.NewTenantUpgrade(
		"create statement_hints table",
		clusterversion.V25_4_AddSystemStatementHintsTable.Version(),
		upgrade.NoPrecondition,
		createStatementHintsTable,
		upgrade.RestoreActionNotRequired(
			"restore for a cluster predating this table can leave it empty",
		),
	),

	newFirstUpgrade(clusterversion.V26_1_Start.Version()),  // Add this line

	// Note: when starting a new release version, the first upgrade (for
	// Vxy_zStart) must be a newFirstUpgrade. Keep this comment at the bottom.
}

6. Update pkg/build/version.txt

Update the version string:

bash
# Before
v25.4.1

# After
v26.1.0-alpha.00000000

7. Update Schema Changer Rules

This is the most complex step involving multiple files.

a) Copy current rules to a new release directory:

bash
cp -r pkg/sql/schemachanger/scplan/internal/rules/current \
      pkg/sql/schemachanger/scplan/internal/rules/release_25_4

b) Update the package name in all files:

bash
find pkg/sql/schemachanger/scplan/internal/rules/release_25_4 -name "*.go" \
     -exec sed -i '' 's/^package current$/package release_25_4/' {} \;

c) Update BUILD.bazel in release_25_4 directory:

Change the library name and import path:

bazel
# Before
go_library(
    name = "current",
    # ...
    importpath = "github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/current",

# After
go_library(
    name = "release_25_4",
    # ...
    importpath = "github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_4",

Change the test target name:

bazel
# Before
go_test(
    name = "current_test",
    # ...
    embed = [":current"],

# After
go_test(
    name = "release_25_4_test",
    # ...
    embed = [":release_25_4"],

d) Update pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go:

Update the version references:

go
// Before
const (
	// rulesVersion version of elements that can be appended to rel rule names.
	rulesVersion = "-25.4"
)

// rulesVersionKey version of elements used by this rule set.
var rulesVersionKey = clusterversion.V25_4

// After
const (
	// rulesVersion version of elements that can be appended to rel rule names.
	rulesVersion = "-26.1"
)

// rulesVersionKey version of elements used by this rule set.
var rulesVersionKey = clusterversion.V26_1

e) Update pkg/sql/schemachanger/scplan/plan.go:

Add import for the new release:

go
import (
	// ...
	"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/current"
	"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_2"
	"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_3"
	"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_4"  // Add this
	// ...
)

Add to rulesForReleases array (around line 158):

go
var rulesForReleases = []rulesForRelease{
	// NB: sort versions in descending order, i.e. newest supported version first.
	{activeVersion: clusterversion.Latest, rulesRegistry: current.GetRegistry()},
	{activeVersion: clusterversion.V25_4, rulesRegistry: release_25_4.GetRegistry()},  // Add this
	{activeVersion: clusterversion.V25_3, rulesRegistry: release_25_3.GetRegistry()},
	{activeVersion: clusterversion.V25_2, rulesRegistry: release_25_2.GetRegistry()},
}

8. Regenerate Files

a) Update Bazel build files:

bash
./dev gen bazel

This updates various BUILD.bazel files across the codebase.

b) Update releases file:

IMPORTANT: The update-releases-file tool does NOT work for this M.1 task. You must manually update pkg/testutils/release/cockroach_releases.yaml.

Manual Update Steps:

When bumping from version X.Y to X.Z (e.g., 26.1 → 26.2):

  1. Do NOT change any existing mappings - Keep all previous entries as-is
  2. Find the latest mapping - For example: "26.1": predecessor: "25.4"
  3. Add a new entry for the new version with the same predecessor:
yaml
"26.1":
  predecessor: "25.4"
"26.2":
  predecessor: "25.4"   # Use same predecessor as 26.1

Why: Both 26.1 and 26.2 are development versions that haven't been released yet. They both should upgrade from the last stable release (25.4). This will be corrected later in M.3 when we add actual release data.

Example for 26.1 → 26.2 bump:

bash
# Before (what exists):
"25.4":
  latest: 25.4.2
  predecessor: "25.3"
"26.1":
  predecessor: "25.4"

# After (what you should have):
"25.4":
  latest: 25.4.2
  predecessor: "25.3"
"26.1":
  predecessor: "25.4"    # KEEP this entry
"26.2":
  predecessor: "25.4"    # ADD this entry

# Verify your changes
tail -10 pkg/testutils/release/cockroach_releases.yaml

Note: In M.3, when the RC is published, the predecessor relationships will be updated to reflect the actual release hierarchy.

c) Regenerate scplan test outputs:

bash
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite

This updates test data in:

  • pkg/sql/schemachanger/scplan/internal/rules/current/testdata/deprules
  • pkg/sql/schemachanger/scplan/internal/rules/release_25_4/testdata/deprules

d) Regenerate CLI test outputs:

bash
./dev test pkg/cli -f DeclarativeRules --rewrite

This updates:

  • pkg/cli/testdata/declarative-rules/invalid_version

e) Regenerate documentation and other generated files:

bash
./dev generate

This updates:

  • docs/generated/settings/settings-for-tenants.txt
  • docs/generated/settings/settings.html

9. Verify Changes

Run tests to ensure everything is working:

bash
# Test version packages
./dev test pkg/clusterversion pkg/roachpb

# Test schema changer
./dev test pkg/sql/schemachanger/scplan/internal/rules/...

# Test CLI
./dev test pkg/cli -f DeclarativeRules

Expected Files Modified

A typical M.1 bump should modify approximately 15-20 files:

Core version files:

  1. pkg/clusterversion/cockroach_versions.go
  2. pkg/roachpb/version.go
  3. pkg/roachpb/version_test.go
  4. pkg/sql/catalog/systemschema/system.go
  5. pkg/upgrade/upgrades/upgrades.go
  6. pkg/build/version.txt

Generated/updated files: 7. pkg/BUILD.bazel 8. pkg/testutils/release/cockroach_releases.yaml 9. pkg/sql/logictest/REPOSITORIES.bzl 10. pkg/cli/testdata/declarative-rules/invalid_version 11. docs/generated/settings/settings-for-tenants.txt 12. docs/generated/settings/settings.html

Schema changer files: 13. pkg/sql/schemachanger/scplan/plan.go 14. pkg/sql/schemachanger/scplan/BUILD.bazel 15. pkg/sql/schemachanger/scplan/internal/rules/current/helpers.go 16. pkg/sql/schemachanger/scplan/internal/rules/current/testdata/deprules 17. pkg/sql/schemachanger/scplan/internal/rules/release_25_4/ (entire new directory)

Common Errors and Solutions

Error: "package release_25_4 not found"

Cause: The new release directory wasn't created or the package name wasn't updated correctly.

Fix:

  1. Verify the directory exists: ls pkg/sql/schemachanger/scplan/internal/rules/release_25_4
  2. Check package names: grep "^package " pkg/sql/schemachanger/scplan/internal/rules/release_25_4/*.go
  3. Re-run the sed command from step 7b if needed

Error: "undefined: clusterversion.V26_1"

Cause: The placeholder constant wasn't added to cockroach_versions.go.

Fix: Add the const V26_1 = Latest line as shown in step 1.

Error: "test output didn't match expected" in scplan tests

Cause: Test outputs need to be regenerated after changing version constants.

Fix: Run the rewrite commands:

bash
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
./dev test pkg/cli -f DeclarativeRules --rewrite

Error: Build failures in Bazel

Cause: BUILD.bazel files weren't regenerated or are out of sync.

Fix:

bash
./dev gen bazel

Error: Release file contains old release

Cause: The releases file wasn't updated or cached data is stale.

Fix:

  1. Clean build: bazel clean
  2. Rebuild and update:
    bash
    bazel build //pkg/cmd/release:release
    _bazel/bin/pkg/cmd/release/release_/release update-releases-file
    

Error: check_generated_code failure in CI

Cause: Generated documentation files weren't updated.

Fix:

bash
./dev generate

Important Notes

  • Do NOT update PreviousRelease - This doesn't happen until M.3
  • MUST manually update pkg/testutils/release/cockroach_releases.yaml - Add new version entry, keep previous entries (see step 8b for details)
  • Version numbers use the previous release's minor for internal versions - V26_1_Start has version 25.4-2, not 26.1-2
  • All internal version numbers must be even - This convention must be maintained
  • Schema changer rules must be versioned - Each release gets its own frozen copy of the rules

Verification Checklist

Before committing, verify:

  • All version constants follow naming conventions (V{MAJOR}_{MINOR}_Start)
  • Internal version number is even (e.g., 2, 4, 6)
  • Successor map includes new version
  • SystemDatabaseSchemaBootstrapVersion points to new start version
  • First upgrade added for new version
  • version.txt updated to new alpha version
  • Schema changer rules copied and updated
  • All tests pass: ./dev test pkg/clusterversion pkg/roachpb pkg/sql/schemachanger/scplan/internal/rules/... pkg/cli -f DeclarativeRules
  • Git status shows expected number of modified files (~15-20)
  • Releases file no longer contains the forked release version

Example PRs

Timeline Context

In the release cycle:

  • Now (M.1): Master bumped to 26.1, release-25.4 exists but no upgrade tests yet
  • M.2 (later): After first 25.4 RC, enable mixed-cluster logic tests
  • M.3 (later): After first 25.4 RC published, enable upgrade tests, update PreviousRelease
  • M.4 (later): Bump MinSupported version
  • M.5 (later): Finalize gates and bootstrap data when 25.4.0 final is released

Common Test Failures After M.1 PR

After the main M.1 PR is merged, CI tests will typically reveal failures that need fixing. CRITICAL: Before modifying any code, use the decision tree below to determine the correct approach.

Decision Tree for Test Failures

When a test fails after M.1 changes, follow this decision tree:

Test fails after M.1 changes
    │
    ├─ FIRST: Did you modify production code beyond the runbook steps?
    │  └─ YES → STOP! Revert changes and consult previous M.1 PRs
    │  └─ NO → Continue below
    │
    ├─ Is it a logic test expecting old version number?
    │  └─ YES → Update test expectations in testdata file (Type 2A, 2B, 2C below)
    │
    ├─ Is it a bootstrap hash mismatch?
    │  └─ YES → Regenerate with -rewrite flag (See Type 2D below)
    │
    ├─ Is it a mixedversion test failure?
    │  └─ Did you modify mixedversion production code?
    │      ├─ YES → REVERT immediately! Check previous M.1 PRs first
    │      └─ NO → Check test history (Type 2E below)
    │
    └─ Is it failing in production code logic?
       └─ Check test history: When was this test added?
           ├─ BEFORE last M.1 → Likely test expectation issue
           └─ AFTER last M.1 → May be new regression test requiring code change (Type 1 below)

⚠️ CRITICAL WARNINGS: When NOT to Modify Production Code

RED FLAGS that indicate you're going the wrong direction:

  • ❌ Adding imports to files that didn't have them (unless it's a new regression test requirement)
  • ❌ Adding conditional logic based on version counts
  • ❌ "Fixing" one test breaks multiple others (ALWAYS WRONG)
  • ❌ Previous M.1 PRs didn't make similar changes (unless new test was added since then)
  • ❌ Adding BUILD.bazel dependencies to mixedversion package

GREEN FLAGS that indicate production code change may be needed:

  • ✅ Failing test was added as regression test AFTER last M.1
  • ✅ Test explicitly checks version-bump edge cases with clear documentation
  • ✅ Test commit message references specific issue number
  • ✅ Multiple related tests fail expecting the same new behavior

Before modifying production code, ALWAYS:

  1. Check: Did previous M.1 PRs modify this file?
    bash
    gh pr view 149494 --json files --jq '.files[] | select(.path == "path/to/file") | .path'
    
  2. Check: Was the failing test added AFTER the last M.1?
    bash
    git log -S "TestName" --oneline --all
    git show <commit-hash>  # Review the test's purpose
    
  3. Verify there's a clear bug, not just making the test pass
  4. Check if test expectation needs updating instead

Type 1: Version Gate Failures (Rare - Requires Code Changes)

These are logic errors in code that references version gates OR new regression tests added after the previous M.1.

How to identify:

  • Test was added AFTER the previous M.1 PR
  • Test explicitly validates version-bump edge cases
  • Test documents a specific issue number it's fixing

Example locations:

  • Version-dependent feature checks
  • Migration logic
  • Upgrade compatibility code
  • Mixedversion test behavior (only if test is new)

Action:

  1. Find when the test was added:
    bash
    git log -S "TestSupportsSkipCurrentVersion" --oneline --all
    
  2. Compare date with last M.1 (e.g., PR #149494 was July 2025)
  3. If test is newer, read the test carefully to understand required behavior
  4. Make minimal production code changes to satisfy the new test
  5. Run full test suite for the package (not just the specific test)

Type 2: Test Expectation Updates (Most Common - No Code Changes)

These are straightforward test output updates that happen because version numbers and URLs changed. Do NOT modify production code for these failures.

A. Version Number Updates in Test Outputs

Pattern: Tests that query version information expect the new version number.

Common files:

  • pkg/sql/logictest/testdata/logic_test/upgrade
  • pkg/sql/logictest/testdata/logic_test/crdb_internal
  • pkg/ccl/logictestccl/testdata/logic_test/crdb_internal_tenant

Changes needed:

diff
 query T
 SELECT crdb_internal.release_series(crdb_internal.node_executable_version())
 ----
-25.4
+26.1

Quick fix:

bash
# Use replace_all=true for files with multiple occurrences
sed -i '' 's/^25\.4$/26.1/' <file>
B. systemDatabaseSchemaVersion Update

Pattern: After bumping to version X.Y, the systemDatabaseSchemaVersion must reflect VX_Y_Start.

File: pkg/sql/logictest/testdata/logic_test/crdb_internal_catalog

How to determine new values:

  1. Find VX_Y_Start in pkg/clusterversion/cockroach_versions.go:

    go
    V26_1_Start: {Major: 25, Minor: 4, Internal: 2}
    
  2. Convert to JSON format:

    json
    {
      "majorVal": 1000025,    // Major * 1000000 + MinorSeries
      "minorVal": 4,          // Minor from VX_Y_Start
      "internal": 2           // Internal from VX_Y_Start
    }
    

Example change:

diff
-"systemDatabaseSchemaVersion": {"internal": 14, "majorVal": 1000025, "minorVal": 3}
+"systemDatabaseSchemaVersion": {"internal": 2, "majorVal": 1000025, "minorVal": 4}

Test to verify:

bash
./dev test pkg/sql/logictest --filter='TestReadCommittedLogic/crdb_internal_catalog'
C. Bootstrap Schema Hash Mismatches

Pattern: Bootstrap hash values change when system schema versions change.

Symptoms:

  • TestInitialValuesToString - hash mismatch
  • TestValidateSystemSchemaAfterBootStrap - schema validation failures

Files typically affected:

  • pkg/sql/catalog/bootstrap/testdata/testdata (hash values)
  • pkg/sql/catalog/systemschema_test/testdata/bootstrap_system
  • pkg/sql/catalog/systemschema_test/testdata/bootstrap_tenant

Fix: Regenerate bootstrap test data

bash
bazel test //pkg/sql/catalog/bootstrap:bootstrap_test \
  --test_arg=-test.v \
  --test_arg=-rewrite \
  --sandbox_writable_path=$(pwd)/pkg/sql/catalog/bootstrap

Verification:

bash
# Run test again WITHOUT -rewrite to confirm it passes
./dev test pkg/sql/catalog/bootstrap -f='TestInitialValuesToString'
D. Mixedversion Test Failures (TRICKY!)

Pattern: Tests show unexpected upgrade paths or version selections.

⚠️ CRITICAL WARNING: These failures are almost NEVER caused by the version bump itself. They're usually caused by incorrect modifications to mixedversion production code.

Common symptoms:

  • TestTestPlanner/step_stages - extra version in upgrade path
  • Test_choosePreviousReleases/skip-version_upgrades - wrong version list
  • TestSupportsSkipUpgradeTo - unexpected true/false value
  • TestSupportsSkipCurrentVersion - unexpected behavior

BEFORE making any changes:

  1. Check if you made ANY changes to pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go
  2. If YES: REVERT THEM IMMEDIATELY (unless the test is new - see below)
  3. Check previous M.1 PRs - did they modify mixedversion code? (Usually NO)
    bash
    gh pr view 149494 --json files --jq '.files[] | select(.path | contains("mixedversion/mixedversion.go"))'
    

When mixedversion code changes ARE needed: Only if ALL of these are true:

  • The failing test was added AFTER the previous M.1 (check with git log -S "TestName")
  • The test explicitly validates version-bump edge cases
  • The test documents a specific issue number

If testdata regeneration is needed (rare):

bash
bazel test //pkg/cmd/roachtest/roachtestutil/mixedversion:mixedversion_test \
  --test_filter=TestTestPlanner \
  --test_arg=-test.v \
  --test_arg=-rewrite \
  --sandbox_writable_path=$(pwd)/pkg/cmd/roachtest/roachtestutil/mixedversion

Verification:

bash
# ALWAYS run full mixedversion test suite, not just filtered tests
./dev test pkg/cmd/roachtest/roachtestutil/mixedversion

If unsure: Do NOT modify mixedversion code. Instead, ask on Slack (#test-eng channel) or file an issue.

Post-Fix Verification Checklist

After fixing test failures:

bash
# Run the tests that previously failed
./dev test pkg/sql/logictest --filter='<test_name>'

# CRITICAL: Run FULL test suites for packages with modified production code
# Do NOT rely on filtered tests alone - they can miss related failures
./dev test pkg/cmd/roachtest/roachtestutil/mixedversion  # If mixedversion code changed
./dev test pkg/sql/pgwire -f='TestPGTest'                # If pgwire tests failed
./dev test pkg/sql/logictest -f='TestLogic'              # After logic test changes

# Verify file count is similar to previous M.1 PRs
# Expected: ~60-65 files changed total (original PR + test fixes)
git diff --stat <base_branch>

# Run broader tests to ensure nothing broke
./dev test pkg/sql/logictest --filter='TestLogic/local/cluster_settings'
./dev test pkg/sql/catalog/bootstrap

Why full test suites matter:

  • Filtered tests like -f='TestSupportsSkipCurrentVersion' can pass while related tests fail
  • Production code changes can have cascading effects on other tests
  • Running the full package test suite catches these issues before CI does

Files That Should/Shouldn't Change in M.1

Files that SHOULD change in M.1:

  • pkg/clusterversion/cockroach_versions.go (version constants)
  • pkg/sql/logictest/testdata/logic_test/* (version expectations)
  • pkg/sql/catalog/bootstrap/testdata/* (bootstrap data)
  • pkg/sql/catalog/systemschema_test/testdata/* (schema expectations)
  • pkg/sql/schemachanger/scplan/internal/rules/release_X_Y/* (new release rules)
  • pkg/testutils/release/cockroach_releases.yaml (releases file)
  • docs/generated/settings/* (generated docs)

Files that should NOT change in M.1 (unless test is new):

  • pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go (production logic)
  • pkg/clusterversion/clusterversion.go (logic, only constants change)
  • ❌ Production code with upgrade/migration logic (unless new regression test)
  • pkg/sql/logictest/REPOSITORIES.bzl (only changes in M.2 after RC)

Files that may change if testdata regeneration is needed:

  • ⚠️ pkg/cmd/roachtest/roachtestutil/mixedversion/testdata/planner/* (if new regression test)
  • ⚠️ pkg/sql/pgwire/testdata/pgtest/* (URL version updates)

Historical Context

  • PR 149494 (25.3 → 25.4): 60 files changed
  • PR 156225 (25.4 → 26.1): 63 files changed (50 original + 14 test fixes)

This consistency helps validate that the changes are complete and correct.

CRITICAL: Validate Changes Before Creating PR

This task is performed every quarter. Before creating the M.1 PR or fixing any test failures, you MUST validate that the changes follow the same pattern as previous quarterly M.1 PRs.

Step 1: Find the Previous M.1 PR

The most recent M.1 PRs (for reference):

  • PR #149494 - clusterversion: move to 25.4 version (July 2025)
  • PR #139387 - clusterversion: move to 25.2 version (January 2025)

Find the PR for the previous quarter:

bash
# Find recent M.1 version bump PRs
gh pr list --search "clusterversion: move to 25" --state merged --limit 10 \
  --json number,title,mergedAt,url | \
  jq -r '.[] | select(.title | test("move to [0-9]+\\.[0-9]+ version")) |
  "\(.number) | \(.title) | \(.mergedAt) | \(.url)"'

Step 2: Compare Files Changed

BEFORE creating the PR, compare your changes against the previous M.1 PR:

bash
# Get files changed in your current work
git diff --name-only <base_branch> | sort > /tmp/current_files.txt

# Get files changed in previous M.1 PR (example: PR #149494)
gh pr view 149494 --json files --jq '.files[].path' | sort > /tmp/previous_files.txt

# Compare the two lists
echo "=== Files ONLY in current PR (investigate these!) ==="
comm -13 /tmp/previous_files.txt /tmp/current_files.txt

echo "=== Files ONLY in previous PR (you might be missing these!) ==="
comm -23 /tmp/previous_files.txt /tmp/current_files.txt

echo "=== Common files (expected) ==="
comm -12 /tmp/previous_files.txt /tmp/current_files.txt

Step 3: Justify Each Unexpected Change

For every file that appears in your PR but NOT in the previous M.1 PR, you must:

  1. Understand why it changed - Read the actual diff
  2. Verify it's intentional - Check if it matches a pattern from the runbook
  3. If it's from release update-releases-file - Compare with previous PR to see if the tool modified this file before
  4. Document or revert - Either justify why it's needed now, or revert the change

Example investigation:

bash
# Check if REPOSITORIES.bzl was modified in previous M.1 PR
gh pr view 149494 --json files --jq '.files[] | select(.path == "pkg/sql/logictest/REPOSITORIES.bzl") | .path'
# If this returns nothing, the file should NOT change in M.1!

# Check what changed in your version
git diff <base_branch> -- pkg/sql/logictest/REPOSITORIES.bzl

Step 4: File-by-File Pattern Validation

For files that appear in BOTH PRs, verify the changes follow the same pattern:

Expected file changes (from runbook and previous PRs):

FileExpected ChangeVerify
pkg/clusterversion/cockroach_versions.goAdd new version constants, update table✓ Always changes
pkg/build/version.txtBump to new alpha version✓ Always changes
pkg/roachpb/version.goUpdate successor series map✓ Always changes
pkg/sql/catalog/systemschema/system.goUpdate bootstrap version✓ Always changes
pkg/upgrade/upgrades/upgrades.goAdd first upgrade✓ Always changes
pkg/sql/schemachanger/scplan/internal/rules/release_X_Y/*New directory with copied rules✓ Always changes
pkg/sql/schemachanger/scplan/plan.goAdd new release to rules map✓ Always changes
pkg/testutils/release/cockroach_releases.yamlUpdate releases (via tool)✓ Always changes
docs/generated/settings/*Regenerated docs✓ Always changes
pkg/sql/catalog/bootstrap/testdata/*Updated bootstrap data✓ Always changes
pkg/BUILD.bazelUpdated build rules✓ Always changes
pkg/sql/logictest/REPOSITORIES.bzlShould NOT change in M.1⚠️ Only changes in M.2

Step 5: Red Flags - Files That Should NOT Change in M.1

Based on comparing with previous M.1 PRs, these files typically should NOT change:

  1. pkg/sql/logictest/REPOSITORIES.bzl

    • Only changes in M.2 (after RC is published)
    • If release update-releases-file modified it, revert the change
    • The 25.4.0-rc.1 config should appear in M.2, not M.1
  2. Test expectation files (before running tests)

    • Test fixes come AFTER the M.1 PR is created and CI runs
    • Don't include test fixes in the initial M.1 commit
  3. Files unrelated to version bumping

    • Check git blame to see if changes are from uncommitted local work

Step 6: Validation Checklist

Before creating the M.1 PR, verify:

  • Compared files changed with previous M.1 PR (#149494 or similar)
  • Investigated every file that's different from the pattern
  • Verified REPOSITORIES.bzl was NOT modified (or reverted if it was)
  • Justified or reverted any unexpected changes
  • File count is similar to previous M.1 PR (~50-60 files for base PR)
  • No test expectation updates in the base PR (those come after CI runs)

Why This Matters

Real example from PR #156225:

  • REPOSITORIES.bzl was incorrectly modified by release update-releases-file
  • It removed 25.2.7 config and added 25.4.0-rc.1 config
  • Previous PR #149494 did NOT modify this file
  • This caused test failures and wasted time fixing tests based on incorrect baseline
  • Had to revert the file to pre-M.1 state

Time saved: If we had validated against PR #149494 before creating the PR, we would have caught this immediately and avoided fixing test failures from a broken baseline.

Quick Reference Commands

bash
# Step 7: Schema changer setup
cp -r pkg/sql/schemachanger/scplan/internal/rules/current \
      pkg/sql/schemachanger/scplan/internal/rules/release_25_4
find pkg/sql/schemachanger/scplan/internal/rules/release_25_4 -name "*.go" \
     -exec sed -i '' 's/^package current$/package release_25_4/' {} \;

# Step 8: Regeneration
./dev gen bazel
bazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
./dev test pkg/sql/schemachanger/scplan/internal/rules/... --rewrite
./dev test pkg/cli -f DeclarativeRules --rewrite
./dev generate

# Verification
./dev test pkg/clusterversion pkg/roachpb
./dev test pkg/sql/schemachanger/scplan/internal/rules/...
./dev test pkg/cli -f DeclarativeRules