pkg/clusterversion/runbooks/M3_enable_upgrade_tests_QUICK.md
Full details: See M3_enable_upgrade_tests.md
Enable upgrade tests after the first RC is published using a 2-PR approach (recommended).
Critical: Fixtures MUST be generated on gceworker (amd64), NOT Mac.
Reference PRs:
git checkout master
git pull origin master
git checkout -b enable-upgrade-tests-25.4-fixtures
# Build and run release tool
bazel build //pkg/cmd/release:release
_bazel/bin/pkg/cmd/release/release_/release update-releases-file
Files updated:
pkg/testutils/release/cockroach_releases.yaml - Adds RC versionpkg/sql/logictest/REPOSITORIES.bzl - Adds RC binaries with checksumsThe tool may incorrectly remove older version binaries still needed by active testserver configs.
# Check active testserver configs
grep "cockroach-go-testserver-" pkg/sql/logictest/logictestbase/logictestbase.go | grep "Name:"
# Verify REPOSITORIES.bzl has binaries for ALL active versions
grep -E "^\s+\(\"25\.[0-9]" pkg/sql/logictest/REPOSITORIES.bzl
If a version is missing:
# Get old config
git show HEAD^:pkg/sql/logictest/REPOSITORIES.bzl | grep -A 4 "25.X.Y"
# Manually restore to REPOSITORIES.bzl
Pattern: Keep N-2, N-1, and N release binaries until their testserver configs are removed.
git add pkg/testutils/release/cockroach_releases.yaml pkg/sql/logictest/REPOSITORIES.bzl
git commit -m "master: Update pkg/testutils/release/cockroach_releases.yaml
Updates releases file with v25.4.0-rc.1 RC and adds RC binaries to REPOSITORIES.bzl.
Part of M.3 fixtures preparation.
Release note: None
Epic: None"
⚠️ MUST run update-firewall BEFORE create, or connection will fail!
# On Mac - set zone (optional)
export CLOUDSDK_COMPUTE_ZONE=us-west1-a # Or us-east1-b
# Update firewall FIRST
./scripts/gceworker.sh update-firewall
# Create gceworker (takes 5-10 min)
./scripts/gceworker.sh create
# SSH to gceworker
./scripts/gceworker.sh start
Zones:
us-west1-a (Oregon - West Coast)us-east1-b (South Carolina - East Coast)On gceworker terminal:
mkdir -p ~/go/src/github.com/cockroachdb
cd ~/go/src/github.com/cockroachdb
git clone [email protected]:<your-username>/cockroach.git
cd cockroach
# Add upstream and fetch tags
git remote add upstream https://github.com/cockroachdb/cockroach.git
git fetch upstream --tags
# Checkout RC tag
git checkout v25.4.0-rc.1
# Set environment
export FIXTURE_VERSION=v25.4.0-rc.1
export COCKROACH_DEV_LICENSE="<your-license>"
# Verify
echo $COCKROACH_DEV_LICENSE
# Configure (first time only)
./dev doctor
# Press Enter for "dev" config
# Type "n" for lintonbuild
# Build
./dev build cockroach short //c-deps:libgeos roachprod workload roachtest
# Clean up previous clusters (error "does not exist" is OK)
./bin/roachprod destroy local
./bin/roachtest run generate-fixtures --local --debug \
--cockroach ./cockroach --suite fixtures
Expected: Test FAILS intentionally and prints move commands.
# Copy command from test output and run it:
for i in 1 2 3 4; do
mkdir -p pkg/cmd/roachtest/fixtures/${i} && \
mv artifacts/generate-fixtures/run_1/logs/${i}.unredacted/checkpoint-*.tgz \
pkg/cmd/roachtest/fixtures/${i}/
done
# Verify
ls -lh pkg/cmd/roachtest/fixtures/*/checkpoint-v25.4.tgz
Expected: 4 files, each ~3-5 MB.
On Mac terminal:
# Create tmp directory
mkdir -p /tmp/fixtures-25.4
# Copy from gceworker (adjust username/zone)
scp -r gceworker-<yourname>.us-west1-a.cockroach-workers:~/go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/fixtures /tmp/fixtures-25.4/
# Navigate to local repo
cd ~/go/src/github.com/cockroachdb/cockroach
git checkout enable-upgrade-tests-25.4-fixtures
# Copy fixtures
for i in 1 2 3 4; do
cp /tmp/fixtures-25.4/fixtures/${i}/checkpoint-v25.4.tgz pkg/cmd/roachtest/fixtures/${i}/
done
# Verify
ls -lh pkg/cmd/roachtest/fixtures/*/checkpoint-v25.4.tgz
git add pkg/cmd/roachtest/fixtures/*/checkpoint-v25.4.tgz
git status # Verify
git commit -m "roachtest: add 25.4 fixtures
Adds roachtest fixtures for v25.4.0-rc.1 to enable upgrade testing.
Fixtures generated on gceworker by running:
./bin/roachtest run generate-fixtures --local --debug \\
--cockroach ./cockroach --suite fixtures
Part of M.3 \"Enable upgrade tests\" checklist.
Release note: None
Epic: None
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>"
# Push to fork
git push -u celiala enable-upgrade-tests-25.4-fixtures
# Create PR
gh pr create --repo cockroachdb/cockroach \
--title "master: Update releases file and add 25.4 fixtures" \
--body "Part of M.3: Enable upgrade tests for 25.4.
This PR:
- Updates releases file with v25.4.0-rc.1
- Updates REPOSITORIES.bzl with RC binaries
- Adds roachtest fixtures for v25.4.0-rc.1
Fixtures generated on gceworker using amd64 architecture.
Epic: None"
Wait for this PR to merge before proceeding to PR 2.
git checkout master
git pull origin master
git checkout -b enable-upgrade-tests-25.4-code
File: pkg/clusterversion/cockroach_versions.go (line ~332)
// Before
const PreviousRelease Key = V25_3
// After
const PreviousRelease Key = V25_4
File: pkg/sql/logictest/logictestbase/logictestbase.go
Add config (after 25.3 config, line ~560):
{
// This config runs tests using a 25.4 predecessor binary, testing upgrade
// compatibility.
Name: "cockroach-go-testserver-25.4",
NumNodes: 1,
OverrideDistSQLMode: "off",
UseCockroachGoTestserver: true,
CockroachGoTestserverVersion: "v25.4.0",
DeclarativeCorpusCollection: true,
},
Add to set (line ~615):
"cockroach-go-testserver-configs": makeConfigSet(
"cockroach-go-testserver-25.2",
"cockroach-go-testserver-25.3",
"cockroach-go-testserver-25.4", // Add this
),
File: pkg/sql/logictest/BUILD.bazel (line ~160)
cockroach_predecessor_version(
name = "cockroach_predecessor_version",
visibility = [
"//pkg/ccl/logictestccl:__subpackages__",
"//pkg/sql/logictest/tests/cockroach-go-testserver-25.2:__pkg__",
"//pkg/sql/logictest/tests/cockroach-go-testserver-25.3:__pkg__",
"//pkg/sql/logictest/tests/cockroach-go-testserver-25.4:__pkg__", # Add
"//pkg/sql/sqlitelogictest:__subpackages__",
],
)
File: pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go (line ~850)
func supportsSkipUpgradeTo(v *version.Version) bool {
return v.Major() == 25 && v.Minor() == 4 && v.Patch() == 0 && v.PreRelease() == ""
}
Check: If Minor() == 4 covers 25.4, no changes needed. Otherwise, update condition.
./dev gen bazel
Generates:
pkg/sql/logictest/tests/cockroach-go-testserver-25.4/BUILD.bazelpkg/sql/logictest/tests/cockroach-go-testserver-25.4/generated_test.gogit add -A
git status # Verify
git commit -m "clusterversion: bump PreviousRelease to V25_4
Updates PreviousRelease constant from V25_3 to V25_4 and adds the
cockroach-go-testserver-25.4 logictest configuration to enable
upgrade tests for version 25.4.
Changes:
- Updated PreviousRelease constant
- Added cockroach-go-testserver-25.4 test configuration
- Generated test files for new config
- Updated BUILD.bazel files via ./dev gen bazel
The supportsSkipUpgradeTo logic already handles 25.4 correctly.
Part of M.3 \"Enable upgrade tests\" checklist.
Release note: None
Epic: None
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>"
# Push to fork
git push -u celiala enable-upgrade-tests-25.4-code
# Create PR
gh pr create --repo cockroachdb/cockroach \
--title "clusterversion: bump PreviousRelease to V25_4" \
--body "Part of M.3: Enable upgrade tests for 25.4.
This PR:
- Updates PreviousRelease constant to V25_4
- Adds cockroach-go-testserver-25.4 logictest configuration
- Generates test files for the new configuration
Depends on fixtures PR being merged first.
Epic: None"
pkg/testutils/release/cockroach_releases.yamlpkg/sql/logictest/REPOSITORIES.bzl
3-6. pkg/cmd/roachtest/fixtures/{1,2,3,4}/checkpoint-v25.4.tgzpkg/clusterversion/cockroach_versions.gopkg/sql/logictest/logictestbase/logictestbase.gopkg/sql/logictest/BUILD.bazelpkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go (verify only)pkg/sql/logictest/tests/cockroach-go-testserver-25.4/BUILD.bazel (generated)pkg/sql/logictest/tests/cockroach-go-testserver-25.4/generated_test.go (generated)
7-9. Various BUILD.bazel updates from ./dev gen bazel# Verify fixture sizes
ls -lh pkg/cmd/roachtest/fixtures/*/checkpoint-v25.4.tgz
# Verify releases file updated
grep -A 2 '"25.4":' pkg/testutils/release/cockroach_releases.yaml
# Verify REPOSITORIES.bzl has RC binaries
grep "25.4.0-rc.1" pkg/sql/logictest/REPOSITORIES.bzl
# Verify PreviousRelease updated
grep "const PreviousRelease" pkg/clusterversion/cockroach_versions.go
# Verify testserver config added
grep "cockroach-go-testserver-25.4" pkg/sql/logictest/logictestbase/logictestbase.go
# Build succeeds
./dev build short
| Warning | Impact | Prevention |
|---|---|---|
| ⚠️ Fixtures on Mac | Wrong architecture, tests fail | MUST use gceworker (amd64) |
| ⚠️ REPOSITORIES.bzl verification | Nightly builds break | Manually verify ALL active testserver versions present |
| ⚠️ gceworker firewall | Can't connect to gceworker | MUST run update-firewall BEFORE create |
| ⚠️ Session disconnect | Lose environment vars | Re-export FIXTURE_VERSION and COCKROACH_DEV_LICENSE |
| Error | Fix |
|---|---|
| SSH connection fails (exit 255) | ./scripts/gceworker.sh start (retry) or destroy & recreate |
| "bazel: command not found" | Run ./dev doctor to install |
| "COCKROACH_DEV_LICENSE not set" | export COCKROACH_DEV_LICENSE="<license>" |
| Error | Fix |
|---|---|
| "license required" | Verify: echo $COCKROACH_DEV_LICENSE |
| "roachprod cluster exists" | ./bin/roachprod destroy local |
| Fixtures 0 bytes or >10MB | Compare with previous version sizes, regenerate if needed |
| Error | Fix |
|---|---|
| Tool removed needed version | `git show HEAD^:pkg/sql/logictest/REPOSITORIES.bzl |
| Nightly build fails "missing binary" | Verify all active testserver versions in REPOSITORIES.bzl |
If you prefer one combined PR (not recommended):
Recommendation: Stick with 2-PR approach for easier review.