pkg/clusterversion/runbooks/M4_bump_minsupported_version.md
This section provides step-by-step instructions for the M.4 task: "Bump MinSupported version" on the master branch. This task removes support for the oldest version in the rolling upgrade window.
📋 Quick Reference: For a streamlined checklist-style guide, see M4_bump_minsupported_version_QUICK.md. This document provides detailed explanations and troubleshooting.
When to perform: After the final release is published (e.g., after v25.2.0 is released, bump MinSupported from v25.2 to v25.4).
What it does:
Why: CockroachDB maintains a rolling upgrade window of N-2 versions. When a new version is released (e.g., 25.4), the oldest version in the window (e.g., 25.2) is no longer supported for direct upgrades. This allows removal of compatibility code and reduces maintenance burden.
Dependencies:
This should be done in 3 PRs:
Note: We often advance MinSupported by two versions (e.g., from v25.2 to v25.4). This is done in two separate PRs to make review easier and reduce the risk of test failures.
Note: We often advance MinSupported up two versions. So this should be done in two PRs:
Each PR follows the same pattern described below.
Before starting, ensure:
master branchThis task should be organized into 5 logical commits based on the reference PR #157767:
Note: Some commits from older reference PRs may not apply to your codebase and should be skipped.
Update the MinSupported constant and all code that references it.
File: pkg/clusterversion/cockroach_versions.go (line ~362)
// Before:
const MinSupported Key = V25_2
// After:
const MinSupported Key = V25_3
Find all files that reference MinSupported:
# Search for files that need updates
git grep -l "MinSupported" pkg/ | grep -v "_test.go" | grep -v "CLAUDE.md"
Common files to update (grep for old version constant like V25_2):
pkg/crosscluster/logical/logical_replication_writer_processor.gopkg/crosscluster/physical/alter_replication_job.gopkg/kv/kvserver/closedts/policyrefresher/policy_refresher.gopkg/kv/kvserver/closedts/sidetransport/sender.gopkg/kv/kvserver/obsolete_code_test.gopkg/sql/backfill/mvcc_index_merger.gopkg/sql/catalog/funcdesc/helpers.gopkg/sql/conn_executor.gopkg/sql/create_index.gopkg/sql/create_table.gopkg/sql/distsql_running.gopkg/storage/pebble.go - Update MinimumSupportedFormatVersion constant⚠️ IMPORTANT: DO NOT modify pkg/sql/execversion/version.go
This file uses a separate versioning mechanism and is owned by the SQL Queries team. It should NOT be modified as part of the M.4 MinSupported bump task. If you see version references in this file during your search, leave them unchanged.
Update each file by changing references from the old MinSupported version to the new one.
CRITICAL: Update pkg/storage/pebble.go (around line 2459):
When bumping MinSupported, you must also update the MinimumSupportedFormatVersion constant to match the Pebble format version for the new MinSupported version.
How to determine the correct value:
pebbleFormatVersionMap in the same file (around line 2450)clusterversion.V25_3)MinimumSupportedFormatVersionExample for bumping MinSupported from V25_2 to V25_3:
// pebbleFormatVersionMap shows:
// V25_3: pebble.FormatValueSeparation
// V25_2: pebble.FormatTableFormatV6 (being removed)
// Therefore, update MinimumSupportedFormatVersion:
-const MinimumSupportedFormatVersion = pebble.FormatTableFormatV6
+const MinimumSupportedFormatVersion = pebble.FormatValueSeparation
Why this is important:
TestMinimumSupportedFormatVersion validates: MinimumSupportedFormatVersion == pebbleFormatVersionMap[MinSupported]expected: 0xNN, actual: 0xMMExample pattern to search for:
# Find files with V25_2 that aren't in testdata
git grep "V25_2" pkg/ | grep -v testdata | grep -v CLAUDE.md | grep -v "_test.go"
Create commit:
git add pkg/clusterversion/cockroach_versions.go pkg/storage/pebble.go <other-modified-files>
git commit -m "clusterversion, storage: bump MinSupported from v25.2 to v25.3
Part of the quarterly M.4 \"Bump MinSupported\" task as outlined in
\`pkg/clusterversion/README.md\`.
This commit updates the MinSupported constant from V25_2 to V25_3,
along with all code that references the MinSupported version.
After this change, clusters running v25.2 can no longer connect to
clusters running master, and direct upgrades from v25.2 to master are
no longer supported.
Changes include updates to:
- Cross-cluster logical and physical replication
- Closed timestamp policy handling
- KV server obsolete code tracking
- SQL backfill and index merging
- Catalog function descriptors
- Connection executor and DDL operations
- Distributed SQL execution versioning
- Storage layer: MinimumSupportedFormatVersion updated to match new MinSupported
Part of #147634 (reference PR for this quarterly task).
Release note: None"
Remove the frozen schema changer rules for the old MinSupported version.
Directory to delete: pkg/sql/schemachanger/scplan/internal/rules/release_25_2/
# Delete the entire directory
rm -rf pkg/sql/schemachanger/scplan/internal/rules/release_25_2/
File: pkg/sql/schemachanger/scplan/plan.go
Remove the import:
// Remove this line:
"github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scplan/internal/rules/release_25_2"
Remove from rulesForReleases array (around line 158):
// Remove this line:
{activeVersion: clusterversion.TODO_Delete_V25_2, rulesRegistry: release_25_2.GetRegistry()},
File: pkg/sql/schemachanger/scplan/BUILD.bazel
Remove the dependency:
# Remove from deps list:
"//pkg/sql/schemachanger/scplan/internal/rules/release_25_2",
Create commit:
git add -A # Captures deletions
git commit -m "schemachanger: remove release_25_2 schema changer rules
Part of the quarterly M.4 \"Bump MinSupported\" task as outlined in
\`pkg/clusterversion/README.md\`.
After bumping MinSupported from v25.2 to v25.3, the frozen schema
changer rules for release 25.2 are no longer needed. These rules were
used to ensure schema changes work correctly in mixed-version clusters
with v25.2 nodes, which are no longer supported.
This commit removes:
- The entire release_25_2 rules directory
- Import and registry entry from plan.go
- Bazel dependency
Part of #147634 (reference PR for this quarterly task).
Release note: None"
Delete the bootstrap data files for the old MinSupported version.
Files to delete:
rm pkg/sql/catalog/bootstrap/data/25_2_system.keys
rm pkg/sql/catalog/bootstrap/data/25_2_system.sha256
rm pkg/sql/catalog/bootstrap/data/25_2_tenant.keys
rm pkg/sql/catalog/bootstrap/data/25_2_tenant.sha256
File: pkg/sql/catalog/bootstrap/initial_values.go
Remove the V25_2 entry from initialValuesFactoryByKey map (around line 66):
// REMOVE this entire block:
clusterversion.TODO_Delete_V25_2: hardCodedInitialValues{
system: v25_2_system_keys,
systemHash: v25_2_system_sha256,
nonSystem: v25_2_tenant_keys,
nonSystemHash: v25_2_tenant_sha256,
}.build,
Remove the go:embed variables (around lines 147-157):
// REMOVE these lines:
//go:embed data/25_2_system.keys
var v25_2_system_keys string
//go:embed data/25_2_system.sha256
var v25_2_system_sha256 string
//go:embed data/25_2_tenant.keys
var v25_2_tenant_keys string
//go:embed data/25_2_tenant.sha256
var v25_2_tenant_sha256 string
File: pkg/sql/catalog/bootstrap/BUILD.bazel
Remove embedsrcs entries (around line 10):
# REMOVE these 4 lines:
"data/25_2_system.keys",
"data/25_2_system.sha256",
"data/25_2_tenant.keys",
"data/25_2_tenant.sha256",
Create commit:
git add -A
git commit -m "bootstrap: remove 25.2 bootstrap data
Part of the quarterly M.4 \"Bump MinSupported\" task as outlined in
\`pkg/clusterversion/README.md\`.
This commit removes the bootstrap data for v25.2, which is now below the
minimum supported version after bumping MinSupported from v25.2 to v25.3.
The bootstrap data files are used to initialize clusters at specific
versions. Since clusters can no longer start at v25.2 (it's below
MinSupported), these files are no longer needed.
Changes:
- Removed 25_2_system.keys and 25_2_system.sha256
- Removed 25_2_tenant.keys and 25_2_tenant.sha256
- Removed V25_2 entry from initialValuesFactoryByKey map in initial_values.go
- Removed go:embed variables for v25.2 bootstrap data
- Updated BUILD.bazel to remove embedsrcs for deleted files
- Updated first_upgrade_test.go error message expectations (descriptor
validation errors now use simpler format after bootstrap data removal)
Part of #147634 (reference PR for this quarterly task).
Release note: None"
Update tests that use the cockroach-go-testserver configuration to reference the new MinSupported version.
Files to update:
# Find mixed_version test files
ls pkg/sql/logictest/testdata/logic_test/mixed_version_*
Common files:
pkg/sql/logictest/testdata/logic_test/mixed_version_charpkg/sql/logictest/testdata/logic_test/mixed_version_citextpkg/sql/logictest/testdata/logic_test/mixed_version_ltreepkg/sql/logictest/testdata/logic_test/mixed_version_partial_statsUpdate LogicTest headers:
For files with single config (char, citext):
-# LogicTest: cockroach-go-testserver-25.2
+# LogicTest: cockroach-go-testserver-25.3
For files with multiple configs (ltree, partial_stats):
-# LogicTest: cockroach-go-testserver-25.2 cockroach-go-testserver-25.3
+# LogicTest: cockroach-go-testserver-25.3
Create commit:
git add pkg/sql/logictest/testdata/logic_test/mixed_version_*
git commit -m "logictest: update mixed_version tests to use 25.3 testserver
Part of the quarterly M.4 \"Bump MinSupported\" task as outlined in
\`pkg/clusterversion/README.md\`.
After bumping MinSupported from v25.2 to v25.3, tests that use the
cockroach-go-testserver predecessor binary configuration need to be
updated to test against v25.3 instead of v25.2.
This commit updates the LogicTest headers for mixed-version tests that
validate feature compatibility across version boundaries:
- mixed_version_char: Tests CHAR type upgrades
- mixed_version_citext: Tests case-insensitive text type upgrades
- mixed_version_ltree: Tests ltree type availability after upgrade
- mixed_version_partial_stats: Tests partial statistics with WHERE clause
For ltree and partial_stats, which previously tested with both 25.2 and
25.3 testservers, the headers now only reference 25.3 since 25.2 is
below MinSupported.
Part of #147634 (reference PR for this quarterly task).
Release note: None"
Remove the local-mixed-X.Y test configuration for the old MinSupported version and all references to it.
File: pkg/sql/logictest/logictestbase/logictestbase.go
Remove the config definition (around lines 501-515):
// REMOVE this entire block:
{
Name: "local-mixed-25.2",
NumNodes: 1,
OverrideDistSQLMode: "off",
BootstrapVersion: clusterversion.TODO_Delete_V25_2,
DisableUpgrade: true,
DeclarativeCorpusCollection: true,
DisableSchemaLockedByDefault: true,
},
Remove from default-configs set (around line 660):
// REMOVE "local-mixed-25.2" from this list:
"default-configs": makeConfigSet(
"local",
// ...
"local-mixed-25.2", // REMOVE THIS LINE
"local-mixed-25.3",
"local-mixed-25.4",
),
Remove from schema-locked-disabled set (around lines 684-686):
// REMOVE "local-mixed-25.2" from this set:
"schema-locked-disabled": makeConfigSet(
"local-legacy-schema-changer",
"local-mixed-25.2", // REMOVE THIS LINE
),
Delete test directories:
rm -rf pkg/ccl/logictestccl/tests/local-mixed-25.2/
rm -rf pkg/sql/logictest/tests/local-mixed-25.2/
rm -rf pkg/sql/sqlitelogictest/tests/local-mixed-25.2/
Update nightly build script:
⚠️ CRITICAL: This file is easy to miss but will break the nightly builds if not updated.
File: build/teamcity/cockroach/nightlies/sqllogic_corpus_nightly_impl.sh
This script runs nightly corpus generation for all mixed version configurations. Update the loop to remove the old version:
# Find the line (around line 84):
for config in local-mixed-25.2 local-mixed-25.3; do
# Update to remove local-mixed-25.2:
for config in local-mixed-25.3; do
Example for bumping from 25.2 to 25.3:
# Before:
for config in local-mixed-25.2 local-mixed-25.3; do
# After:
for config in local-mixed-25.3; do
Example for bumping from 25.3 to 25.4:
# Before:
for config in local-mixed-25.3 local-mixed-25.4; do
# After:
for config in local-mixed-25.4; do
Why this matters:
local-mixed-X.Y directoryRemove references from test files:
⚠️ CRITICAL: Understanding onlyif and skipif Directives
The onlyif config and skipif config directives in logic tests guard the next statement(s):
onlyif config local-mixed-25.2 → Run the next statement ONLY on local-mixed-25.2skipif config local-mixed-25.2 → Skip the next statement on local-mixed-25.2IMPORTANT: These two directives require DIFFERENT handling when removing a config:
Rule 1: For onlyif config local-mixed-25.2 directives:
Rule 2: For skipif config local-mixed-25.2 directives:
Examples:
# Example 1: onlyif directive (remove both directive AND statement)
# BEFORE:
onlyif config local-mixed-25.2
statement error pgcode 0A000 pq: unimplemented: usage of user-defined function
CREATE TABLE t1(a INT PRIMARY KEY, b INT DEFAULT f1());
statement ok
CREATE TABLE t1(a INT PRIMARY KEY, b INT DEFAULT f1());
# AFTER (CORRECT - removed both directive and guarded statement):
statement ok
CREATE TABLE t1(a INT PRIMARY KEY, b INT DEFAULT f1());
# Example 2: skipif directive (remove ONLY directive, keep statement)
# BEFORE:
skipif config local-mixed-25.2
statement error pgcode 0A000 unimplemented: cannot evaluate function in this context
ALTER TABLE test_tbl_t ADD COLUMN c int AS (test_tbl_f()) stored;
# AFTER (CORRECT - removed only directive, kept statement):
statement error pgcode 0A000 unimplemented: cannot evaluate function in this context
ALTER TABLE test_tbl_t ADD COLUMN c int AS (test_tbl_f()) stored;
Why the difference?
onlyif: The guarded statement was ONLY for that specific config. Removing the config means we don't need that statement anymore (there's usually an unguarded version later)skipif: The guarded statement was for ALL configs EXCEPT the one being removed. Now that we're removing that config, the statement should run everywherePattern to follow for onlyif:
onlyif config local-mixed-25.2 linePattern to follow for skipif:
skipif config local-mixed-25.2 lineFind all files with local-mixed-25.2 references:
grep -r "local-mixed-25\.2" pkg/sql/logictest/testdata/logic_test/ \
pkg/ccl/logictestccl/testdata/logic_test/ | cut -d: -f1 | sort -u
For each file, remove local-mixed-25.2 using targeted sed commands:
# For files with LogicTest headers, skipif, or onlyif on dedicated lines:
sed -i '' \
-e '/^# LogicTest:/s/ !*local-mixed-25\.2//g' \
-e '/^skipif config local-mixed-25\.2$/d' \
-e '/^onlyif config local-mixed-25\.2$/d' \
"$file"
# For files with multi-config onlyif/skipif lines:
sed -i '' \
-e '/^onlyif config/s/ local-mixed-25\.2//g' \
-e '/^skipif config/s/ local-mixed-25\.2//g' \
"$file"
⚠️ IMPORTANT: Manual review required after sed
The sed commands above only remove the directives, not the statements they guard. You MUST:
onlyif config local-mixed-25.2skipif config local-mixed-25.2, the sed commands are sufficient - no manual work neededHow to find files needing manual review (onlyif only):
# Find files that had onlyif with local-mixed-25.2 (these need manual review)
git diff pkg/sql/logictest/testdata/logic_test/ pkg/ccl/logictestccl/testdata/logic_test/ | \
grep -B 2 "^-onlyif config.*local-mixed-25\.2" | \
grep "^diff --git" | sed 's/.*b\///' | sort -u
# Files with skipif don't need manual review - the directive removal is sufficient
Remove empty LogicTest directive lines:
If any files have headers like # LogicTest: !local-mixed-25.2 that become empty # LogicTest:, remove them:
# Find files with empty LogicTest directives
grep -l "^# LogicTest:$" pkg/sql/logictest/testdata/logic_test/* \
pkg/ccl/logictestccl/testdata/logic_test/*
# Remove the empty lines
sed -i '' '/^# LogicTest:$/d' <file>
Regenerate Bazel files:
./dev gen bazel
Verify the changes:
# Should succeed
./dev build short
Create commit:
git add -A
git commit -m "logictest: remove local-mixed-25.2 test configuration
Part of the quarterly M.4 \"Bump MinSupported\" task as outlined in
\`pkg/clusterversion/README.md\`.
After bumping MinSupported from v25.2 to v25.3, the local-mixed-25.2
test configuration is no longer needed since it simulates a mixed-version
cluster with v25.2 nodes, which can no longer connect to the cluster.
This commit:
- Removes the local-mixed-25.2 config from logictestbase.go
- Removes it from the default-configs and schema-locked-disabled sets
- Deletes the generated test directories for local-mixed-25.2
- Removes all references from logic test files (skipif/onlyif directives)
- Removes empty LogicTest directive lines that resulted from deletions
- Regenerates Bazel BUILD files via \`./dev gen bazel\`
Changes affect 34 test files that had skipif or onlyif directives
referencing local-mixed-25.2, plus the generated test files and BUILD
files that were auto-generated based on the removed configuration.
Part of #147634 (reference PR for this quarterly task).
Release note: None"
A typical M.4 bump should modify approximately 70-80 files across the 6 commits:
Commit 1 (1 file):
pkg/clusterversion/cockroach_versions.go - Prefix version keysCommit 2 (~13 files):
pkg/clusterversion/cockroach_versions.go - MinSupported constantpkg/crosscluster/logical/logical_replication_writer_processor.gopkg/crosscluster/physical/alter_replication_job.gopkg/kv/kvserver/closedts/policyrefresher/policy_refresher.gopkg/kv/kvserver/closedts/sidetransport/sender.gopkg/kv/kvserver/obsolete_code_test.gopkg/sql/backfill/mvcc_index_merger.gopkg/sql/catalog/funcdesc/helpers.gopkg/sql/conn_executor.gopkg/sql/create_index.gopkg/sql/create_table.gopkg/sql/distsql_running.gopkg/sql/execversion/version.gopkg/storage/pebble.go - MinimumSupportedFormatVersion constantCommit 4 (~30 files):
pkg/sql/schemachanger/scplan/internal/rules/release_25_2/ - Entire directory deleted (~25 files)pkg/sql/schemachanger/scplan/plan.gopkg/sql/schemachanger/scplan/BUILD.bazelCommit 5 (6 files):
pkg/sql/catalog/bootstrap/data/25_2_system.keys - Deletedpkg/sql/catalog/bootstrap/data/25_2_system.sha256 - Deletedpkg/sql/catalog/bootstrap/data/25_2_tenant.keys - Deletedpkg/sql/catalog/bootstrap/data/25_2_tenant.sha256 - Deletedpkg/sql/catalog/bootstrap/initial_values.gopkg/sql/catalog/bootstrap/BUILD.bazelCommit 6 (4 files):
pkg/sql/logictest/testdata/logic_test/mixed_version_charpkg/sql/logictest/testdata/logic_test/mixed_version_citextpkg/sql/logictest/testdata/logic_test/mixed_version_ltreepkg/sql/logictest/testdata/logic_test/mixed_version_partial_statsCommit 7 (~65 files):
pkg/sql/logictest/logictestbase/logictestbase.go - Config removalbuild/teamcity/cockroach/nightlies/sqllogic_corpus_nightly_impl.sh - CRITICAL: Update nightly build looppkg/ccl/logictestccl/tests/local-mixed-25.2/BUILD.bazel - Deletedpkg/ccl/logictestccl/tests/local-mixed-25.2/generated_test.go - Deletedpkg/sql/logictest/tests/local-mixed-25.2/BUILD.bazel - Deletedpkg/sql/logictest/tests/local-mixed-25.2/generated_test.go - Deletedpkg/sql/sqlitelogictest/tests/local-mixed-25.2/BUILD.bazel - Deletedpkg/sql/sqlitelogictest/tests/local-mixed-25.2/generated_test.go - Deleted./dev gen bazelpkg/BUILD.bazel - Binary file updated1. Run core tests:
./dev test pkg/clusterversion pkg/storage
Expected: All tests should pass.
2. Verify build:
./dev build short
Expected: Build completes successfully.
3. Check commit structure:
git log --oneline HEAD~6..HEAD
Expected: Should show 6 commits in the correct order.
4. Verify file counts:
git diff --stat <base-branch>
Expected: Approximately 70-80 files changed, with significant deletions (~5,000-6,000 lines).
This task is performed every quarter. Before creating the PR, validate that changes follow the same pattern as the reference PR.
Step 1: Compare file lists
# Get files from reference PR #147634
gh pr view 147634 --json files --jq '.files[].path' | sort > /tmp/ref_m4_files.txt
# Get your current files
git diff --name-only <base-branch> | sort > /tmp/current_m4_files.txt
# Compare
echo "=== Files ONLY in current PR (investigate!) ==="
comm -13 /tmp/ref_m4_files.txt /tmp/current_m4_files.txt
echo "=== Files ONLY in reference PR (might be missing!) ==="
comm -23 /tmp/ref_m4_files.txt /tmp/current_m4_files.txt
Step 2: Justify differences
For each file that appears in your PR but NOT in the reference PR:
Step 3: Verify commit messages match pattern
# Check that commit messages follow the established pattern
git log --format="%s" HEAD~6..HEAD
Expected format:
clusterversion: prefix version keys below 25.3 with TODO_Delete_clusterversion: bump MinSupported from v25.2 to v25.3schemachanger: remove release_25_2 schema changer rulesbootstrap: remove 25.2 bootstrap datalogictest: update mixed_version tests to use 25.3 testserverlogictest: remove local-mixed-25.2 test configurationError: panic: unknown config name local-mixed-25.2
Cause: Test files still reference local-mixed-25.2 after the config was removed from logictestbase.go.
Fix:
# Find all references
grep -r "local-mixed-25\.2" pkg/sql/logictest/testdata/logic_test/ \
pkg/ccl/logictestccl/testdata/logic_test/
# Remove them with sed (see Commit 7 instructions above)
Error: File corruption where lines are concatenated (e.g., # LogicTest: !local-legacy-schema-changer# A basic sanity check...)
Cause: Using overly aggressive perl or sed commands like perl -pi -e 's/\s*local-mixed-25\.2\s*/ /g; s/\s+$//; s/\s+/ /g'
Fix: Don't use global whitespace replacement. Use targeted sed:
# WRONG (removes newlines):
perl -pi -e 's/\s*local-mixed-25\.2\s*/ /g; s/\s+$//; s/\s+/ /g'
# RIGHT (preserves structure):
sed -i '' \
-e '/^# LogicTest:/s/ !*local-mixed-25\.2//g' \
-e '/^skipif config local-mixed-25\.2$/d' \
-e '/^onlyif config local-mixed-25\.2$/d'
Error: empty LogicTest directive during bazel generation
Cause: Files had headers like # LogicTest: !local-mixed-25.2 which became # LogicTest: after removal.
Fix:
# Find files with empty directives
grep -l "^# LogicTest:$" pkg/sql/logictest/testdata/logic_test/* \
pkg/ccl/logictestccl/testdata/logic_test/*
# Remove the empty lines
sed -i '' '/^# LogicTest:$/d' <files>
Error: Nightly build fails with: references nonexistent local-mixed-X.Y logictest directory
Cause: Forgot to update build/teamcity/cockroach/nightlies/sqllogic_corpus_nightly_impl.sh to remove the old config from the loop.
Real-world occurrence: This broke the nightly in PR #158225, requiring follow-up fix in commit 4bc710b4667 and issue #158741.
Fix:
# Edit build/teamcity/cockroach/nightlies/sqllogic_corpus_nightly_impl.sh
# Find the line (around line 84):
for config in local-mixed-25.2 local-mixed-25.3; do
# Update to:
for config in local-mixed-25.3; do
Prevention:
Error: Local files (.claude/, PLAN_*.md, etc.) appear in commit.
Cause: Used git add -A without excluding local files.
Fix:
# Reset the commit
git reset HEAD~ --soft
# Unstage local files
git reset HEAD .claude/settings.local.json PLAN_*.md *.local.md
# Re-commit without local files
git commit -m "..."
Error: Deleted directories don't appear in commit.
Cause: Forgot to use git add -A or git add -u which capture deletions.
Fix:
# Always use git add -A for commits with deletions
git add -A # Adds modifications AND deletions
Error: Test failures like:
expected success, but found
(42P07) relation "test.public.t1" already exists
Cause: Removed onlyif config local-mixed-25.2 directives but forgot to remove the statements they guarded, resulting in duplicate statements that both execute.
Example from udf_in_table:
# File had two CREATE TABLE statements:
# 1. Guarded by "onlyif config local-mixed-25.2" (lines 88-98)
# 2. Unguarded default version (lines 533-547)
# After removing ONLY the directive (WRONG):
statement ok
CREATE TABLE t1(...) # ← Was guarded, now runs on all configs!
# Later:
statement ok
CREATE TABLE t1(...) # ← Unguarded, always runs → DUPLICATE!
How to identify:
# After removing directives with sed, search git diff for removed onlyif lines
git diff pkg/sql/logictest/testdata/logic_test/ | grep "^-onlyif config.*local-mixed-25\.2"
# For each file, manually check if guarded statements should also be removed
Fix:
onlyif directive was guardingExample fix for udf_in_table:
# Remove the entire guarded statement block (lines 88-98)
# Keep only the unguarded version (lines 533-547)
Prevention:
onlyif linesonlyif to check for guarded statements./dev testlogicError: After rebasing against master, pkg/storage/pebble.go has the wrong pebble format version for V25_3.
Symptom:
// WRONG - V25_3 value changed when it shouldn't have:
var pebbleFormatVersionMap = map[clusterversion.Key]pebble.FormatMajorVersion{
clusterversion.V25_4_PebbleFormatV2BlobFiles: pebble.FormatV2BlobFiles,
clusterversion.V25_3: pebble.FormatTableFormatV6, // ← WRONG!
}
// CORRECT - Only removed V25_2, kept V25_3 unchanged:
var pebbleFormatVersionMap = map[clusterversion.Key]pebble.FormatMajorVersion{
clusterversion.V25_4_PebbleFormatV2BlobFiles: pebble.FormatV2BlobFiles,
clusterversion.V25_3: pebble.FormatValueSeparation, // ← CORRECT
}
Cause: During rebase, git may auto-merge changes to pebbleFormatVersionMap incorrectly. The commit that removes local-mixed-25.2 should only remove the V25_2 entry from this map, not change the V25_3 value.
How to identify:
# Check the pebble format version in the logictest removal commit
git show <commit-sha>:pkg/storage/pebble.go | grep -A 4 "pebbleFormatVersionMap ="
# Compare with master to see what the correct V25_3 value should be
git show master:pkg/storage/pebble.go | grep -A 4 "pebbleFormatVersionMap ="
Fix:
# Start interactive rebase to edit the problematic commit
git rebase -i <base-commit>
# Mark the commit with pebble.go for 'edit'
# Then fix the file:
# Edit pkg/storage/pebble.go to restore correct V25_3 value
git add pkg/storage/pebble.go
git commit --amend --no-edit
git rebase --continue
Prevention:
pkg/storage/pebble.go changes carefullyCommit 1 - Bump MinSupported:
# Find files to update
git grep -l "MinSupported" pkg/ | grep -v "_test.go"
git grep "V25_2" pkg/ | grep -v testdata
# Update files manually (including pkg/storage/pebble.go!)
# In pkg/storage/pebble.go, update MinimumSupportedFormatVersion to match
# the pebbleFormatVersionMap entry for the new MinSupported version
git add pkg/clusterversion/cockroach_versions.go pkg/storage/pebble.go <other-files>
git commit -m "clusterversion, storage: bump MinSupported from v25.2 to v25.3..."
# Verify with test
./dev test pkg/storage -f TestMinimumSupportedFormatVersion -v
Commit 2 - Remove schema changer rules:
rm -rf pkg/sql/schemachanger/scplan/internal/rules/release_25_2/
# Edit plan.go and BUILD.bazel
git add -A
git commit -m "schemachanger: remove release_25_2 schema changer rules..."
Commit 3 - Remove bootstrap data:
rm pkg/sql/catalog/bootstrap/data/25_2_*
# Edit initial_values.go and BUILD.bazel
git add -A
git commit -m "bootstrap: remove 25.2 bootstrap data..."
Commit 4 - Update mixed_version tests:
# Edit 4 mixed_version test files
git add pkg/sql/logictest/testdata/logic_test/mixed_version_*
git commit -m "logictest: update mixed_version tests to use 25.3 testserver..."
Commit 5 - Remove local-mixed config:
# Edit logictestbase.go to remove the config definition and references
# CRITICAL: Update nightly build script
# Edit build/teamcity/cockroach/nightlies/sqllogic_corpus_nightly_impl.sh
# Change: for config in local-mixed-25.2 local-mixed-25.3; do
# To: for config in local-mixed-25.3; do
rm -rf pkg/ccl/logictestccl/tests/local-mixed-25.2/
rm -rf pkg/sql/logictest/tests/local-mixed-25.2/
rm -rf pkg/sql/sqlitelogictest/tests/local-mixed-25.2/
# Remove references from test files
find pkg/sql/logictest/testdata/logic_test/ \
pkg/ccl/logictestccl/testdata/logic_test/ \
-type f -exec grep -l "local-mixed-25\.2" {} \; | while read file; do
sed -i '' \
-e '/^# LogicTest:/s/ !*local-mixed-25\.2//g' \
-e '/^skipif config local-mixed-25\.2$/d' \
-e '/^onlyif config local-mixed-25\.2$/d' \
"$file"
done
# Handle multi-config lines
find pkg/sql/logictest/testdata/logic_test/ \
pkg/ccl/logictestccl/testdata/logic_test/ \
-type f -exec grep -l "local-mixed-25\.2" {} \; | while read file; do
sed -i '' \
-e '/^onlyif config/s/ local-mixed-25\.2//g' \
-e '/^skipif config/s/ local-mixed-25\.2//g' \
"$file"
done
# Remove empty LogicTest directives
sed -i '' '/^# LogicTest:$/d' pkg/sql/logictest/testdata/logic_test/udf_in_index \
pkg/ccl/logictestccl/testdata/logic_test/provisioning
./dev gen bazel
git add -A
git commit -m "logictest: remove local-mixed-25.2 test configuration..."
Verification:
# Test
./dev test pkg/clusterversion pkg/storage
# Build
./dev build short
# View commits
git log --oneline HEAD~5..HEAD
In the release cycle:
This is a follow-up task after the main MinSupported bump PRs (e.g., after bumping from v25.2 → v25.3 → v25.4). It marks obsolete version keys for future removal by prefixing them with TODO_Delete_.
When to perform: After the MinSupported bump PRs are merged (e.g., after bumping from V25_2 to V25_4 via two PRs).
What it does:
TODO_Delete_Why separate PR: This change can be large (touching many files) and is purely cosmetic/organizational, so it's cleaner to separate it from the functional MinSupported bump.
Before starting, ensure:
master branchFile: pkg/clusterversion/cockroach_versions.go
Find all INTERNAL version keys below the new MinSupported (e.g., all V25_2_* and V25_3_* keys when MinSupported is now V25_4):
# Example for MinSupported bumped from V25_2 to V25_4 (via V25_3)
# Find all V25_2_* and V25_3_* version keys (NOT V25_2 or V25_3 themselves):
grep -n "^\s*V25_2_\|^\s*V25_3_" pkg/clusterversion/cockroach_versions.go
Add TODO_Delete_ prefix:
// Before:
V25_2_Start
V25_2_AddSystemStatementHintsTable
V25_2 // ← DO NOT PREFIX THIS ONE!
V25_3_Start
V25_3_AddEventLogColumnAndIndex
V25_3_AddEstimatedLastLoginTime
V25_3_AddHotRangeLoggerJob
V25_3 // ← DO NOT PREFIX THIS ONE!
// After:
TODO_Delete_V25_2_Start
TODO_Delete_V25_2_AddSystemStatementHintsTable
V25_2 // ← Keep as-is, no TODO_Delete_ prefix
TODO_Delete_V25_3_Start
TODO_Delete_V25_3_AddEventLogColumnAndIndex
TODO_Delete_V25_3_AddEstimatedLastLoginTime
TODO_Delete_V25_3_AddHotRangeLoggerJob
V25_3 // ← Keep as-is, no TODO_Delete_ prefix
⚠️ IMPORTANT: Do NOT prefix the final release keys (e.g., V25_2, V25_3). Only prefix the internal version keys (e.g., V25_2_Start, V25_3_FeatureName).
Why: The final release keys are referenced in other parts of the codebase and tooling, and they serve as anchors for release tracking.
Find all files that reference the old internal version keys:
# Example for V25_2_* and V25_3_* keys
git grep "V25_2_Start\|V25_2_AddSystemStatementHintsTable" pkg/
git grep "V25_3_Start\|V25_3_AddEventLogColumnAndIndex\|V25_3_AddEstimatedLastLoginTime\|V25_3_AddHotRangeLoggerJob" pkg/
For each file found, update the references:
// Before:
if version.IsActive(clusterversion.V25_3_Start) {
// After:
if version.IsActive(clusterversion.TODO_Delete_V25_3_Start) {
Common files to check:
pkg/upgrade/upgrades/IsActive() for these versionsgit add -A
git commit -m "clusterversion: prefix version keys below 25.4 with TODO_Delete_
Part of the quarterly M.4 \"Bump MinSupported\" task (part 3) as outlined
in \`pkg/clusterversion/README.md\`.
This commit marks version keys below v25.4 with the TODO_Delete_ prefix
to indicate they will be removed in the next major release.
After bumping MinSupported from v25.2 to v25.4, all version gates for
v25.2 and v25.3 features are always active and can be simplified. This
prefix helps track which version checks are obsolete.
Part of the cleanup following the MinSupported bump.
Release note: None"
# Ensure everything still compiles
./dev build short
# Run version tests
./dev test pkg/clusterversion -v
A typical TODO_Delete_ prefix PR should modify:
pkg/clusterversion/cockroach_versions.go (version key definitions)pkg/upgrade/upgrades/ (migration references)Typically 5-20 files total, depending on how many internal version keys existed for those releases.
Before creating the PR:
./dev build short./dev test pkg/clusterversionAfter this PR is merged, create GitHub issues for cleaning up TODO_Delete_ version gates:
Tasks: