Back to Rustfs

CI gate matrix

docs/testing/ci-gates.md

1.0.0-rc.4-preview.111.3 KB
Original Source

CI gate matrix

This file is the source of truth for which validation runs on each event, its configured wall-clock budget, and whether it can block a merge. Test taxonomy, naming, and nextest serialization rules remain in README.md; e2e membership and counts remain in e2e-suite-inventory.md.

The distinction between required and report-only is load-bearing: a failing job blocks a merge only when its exact check name is present in the live main ruleset. A workflow name, a merge_group trigger, or a red PR check does not make a job required by itself.

Required merge checks

The live main ruleset (6436880) currently requires exactly these contexts:

Required contextProducerValidation
CLA Check.github/workflows/cla.ymlContributor agreement
Quick Checks.github/workflows/ci.ymlFormatting and repository guard scripts
Test and Lint.github/workflows/ci.ymlClippy, workspace nextest excluding e2e_test, doctests, and migration proofs

For pull requests limited to the paths excluded by the main CI workflow, .github/workflows/ci-docs-only.yml reports Quick Checks and Test and Lint under the same names. It runs the real quick checks and the planning-document guard; it does not claim that Rust compilation or runtime tests ran. Despite the workflow name, these paths also include selected deploy, workflow, and lock files.

Verify the live rule rather than trusting this snapshot before changing merge policy:

bash
gh api repos/rustfs/rustfs/rulesets/6436880 \
  --jq '.rules[] | select(.type == "required_status_checks") | .parameters'

The ruleset currently has strict_required_status_checks_policy=false. Continuous Integration accepts merge_group events and runs e2e-full for them, but End-to-End Tests (full merge gate) is not currently a required context. Therefore the repository is prepared to test a merge-queue SHA, but the workflow alone does not prove that every merge passed that lane.

Pull request and merge matrix

Budgets below are job timeout-minutes, not typical runtimes. “Report-only” means the result is visible and actionable but is not in the live required context list.

EventValidationBudgetMerge statusReproduction
PR, non-doc changeQuick Checks10 minRequiredmake pre-commit (broader local umbrella)
PR, non-doc changeTest and Lint90 minRequiredcargo nextest run --profile ci --all --exclude e2e_test
PR, non-doc changeTypos10 minReport-onlytypos
PR, non-doc changeILM Integration (serial)90 minReport-onlyUse the exact command in .github/workflows/ci.yml
PR, non-doc changerio-v2 / swift / sftp test-and-lint variants90 min eachReport-onlycargo nextest run with the workflow's feature set
PR, non-doc changeBuild RustFS Debug Binary30 minReport-only; prerequisite for black-box lanescargo build -p rustfs --bins
PR, non-doc changeio_uring Integration (real)30 minReport-onlycargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
PR, non-doc changeEnd-to-End Tests (e2e-smoke plus s3s-e2e)30 minReport-onlycargo nextest run --profile e2e-smoke -p e2e_test; then ./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>
PR, non-doc changeS3 Implemented Tests60 minReport-onlyBuild rustfs, then run scripts/s3-tests/run.sh with DEPLOY_MODE=binary, TEST_MODE=single, and MAXFAIL=0
PR, non-doc changeS3 Lifecycle Behavior Tests30 minReport-onlyUse the accelerated scanner environment in .github/workflows/ci.yml with scripts/s3-tests/run.sh
PR touching dependency or workflow inputsCargo Deny / Workflow Pin Report / Dependency Review20 / 5 / 30 minReport-onlycargo deny check; scripts/security/check_workflow_pins.sh
PR touching architecture rules or architecture docsArchitecture Migration Rules10 minReport-onlyscripts/check_architecture_migration_rules.sh
PR touching Nix or workspace manifestsNix Build & Check60 minReport-onlynix flake check
PR limited to main-CI-excluded pathscompanion Quick Checks and Test and Lint10 min eachRequiredgit diff --check; make doc-paths-check when documentation paths changed
merge_groupStandard CI plus e2e-full55 min for e2e-fullStandard required contexts only; e2e-full report-onlycargo nextest run --profile e2e-full -p e2e_test
Push to mainStandard CI plus e2e-full55 min for e2e-fullPost-merge detectionSame as merge_group
PR touching fuzz inputs or harness pathsBuild plus five 60-second fuzz smoke targets60 min build; 30 min per targetReport-onlyMAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh
PR touching selected ecstore disk/format pathsRename Safety on Windows60 minReport-onlyRun the four cargo test -p rustfs-ecstore --lib <filter> commands in windows-filesystem.yml on Windows

The authoritative e2e filters live in .config/nextest.toml; extend a profile instead of adding a second ad-hoc selector. Before a profile runs, scripts/check_test_wiring.py compares its exact membership to the committed digest so a silent test drop fails closed.

Scheduled and manual validation

Scheduled lanes are independent fault domains. They do not block a pull request, but their workflow-local gate can fail the run and scheduled failures are routed to the shared failure-issue action. The scheduled-validation watchdog and freshness workflow separately detect incomplete runs and missing schedules.

Cadence (UTC unless noted)Workflow / validationBudgetVerdict and artifactsReproduction
Daily 02:17Fuzz: five nightly corpus targets60 min build; 60 min per targetGate; corpus/crash artifacts, scheduled failure alertMAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh
Dormant (cron 03:17 once re-enabled)MinIO interop (EC + SSE read parity)40 minManually disabled in the Actions settings (backlog#1603) and therefore outside the freshness list; re-add it to .github/scheduled-validations.json when re-enablingFollow the pinned Docker fixture steps in minio-interop.yml
Daily 04:29Replication / cluster-fault / protocol e2e45 / 90 / 90 minThree independent gates; JUnit, membership, and server logscargo nextest run --profile e2e-repl-nightly -p e2e_test; --profile e2e-nightly; -j 1 --profile e2e-protocols
Daily 06:31Warp performance A/B180 minRegression budget gate; A/B summaries and server logsbash scripts/run_hotpath_warp_abba.sh --help
Daily 00:07 Asia/Shanghai (16:07 UTC previous day)Nightly GNU build and Vault lanes150 / 90 / 60 minBuild, live Vault, and HA failover gatesUse the commands and pinned Vault images in nightly-gnu.yml
Daily 03:23Security Audit20 / 5 min, plus 30 min on PR dependency reviewCargo Deny and workflow-pin gates; scheduled failure alertcargo deny check; scripts/security/check_workflow_pins.sh
Daily 23:47Scheduled Validation Freshness10 minFails when a critical schedule was never created or is stale; an entry may carry never_ran_grace_until to cover the window before a newly enabled cron's first slotDispatch scheduled-validation-freshness.yml
Sunday 00:11Full Continuous Integration matrixPer-job budgets aboveWeekly variant coverage, including dormant rio-v2 binary/e2e lanesDispatch ci.yml
Sunday 01:13Seven-platform build matrix150 min per platformBuild/package integrity; scheduled failure alertDispatch build.yml with an exact platform set
Sunday 02:19Ceph s3-tests full sweep: single and real four-node, four shards each180 min per shardCompatibility gate; report, JUnit, exact node IDs, and server logsscripts/s3-tests/run.sh against an existing single or distributed target
Sunday 06:41Mint120 minReport-only by design; per-suite PASS/FAIL/NA and raw log.jsonReproduce the pinned Docker sequence in mint.yml or dispatch it
Sunday 07:43Workspace line coverage120 minReport-only trend; lcov and JSON retained 90 daysmake coverage
Monthly, day 1 06:37Runner Hygiene15 minValidates runner ephemerality; scheduled failure alertDispatch runner-hygiene.yml

Manual workflow_dispatch exists for the scheduled workflows above. Manual runs are debugging evidence and intentionally do not open scheduled-failure issues. A manual performance run may explicitly allow a known regression; that override must not be treated as an ordinary passing baseline.

Release validation

Release validation is post-merge and tag-driven; it does not substitute for a pull-request gate.

EventValidationBudgetResult
Push to main or weekly scheduleBuild and Release platform matrix150 min per platformBuild artifacts for all selected targets; no release publication on a main push
Valid release or preview tagBuild and Release plus asset checks150 min per platformDraft release, checksummed assets, and publish step
Successful non-preview release-tag buildDocker image build and image scan60 min build; 30 min scanMulti-architecture images plus vulnerability report
Successful release-tag buildDEB/RPM packaging30 min per architecturePackages and checksum files uploaded to the release
Successful non-preview release-tag buildHelm template test and package30 min build; 30 min publishVersioned chart and repository index

Use an exact preview tag for end-to-end release rehearsal. Manual dispatches are backfill/debug paths and do not prove the automatic workflow_run chain.

A preview Release is internal validation state, not a deliverable: after the final tag's release is published, cleanup-preview-releases deletes every <target>-preview.<N> Release for that target. The tags themselves are kept, so the validated commit stays traceable.

Evidence requirements

A green check is useful only when it proves the intended behavior ran:

  • Record the exact commit SHA and run URL.
  • Separate product failure from runner prerequisites, service readiness, and cancellation. Repair the precondition, then rerun the exact workload.
  • Preserve membership manifests, JUnit, raw compatibility logs, seeds, and server logs where the workflow provides them.
  • For a bug fix or a new fault checker, provide sensitivity evidence: the old behavior or an intentional mutation must fail the new oracle, and the fixed behavior must pass it.
  • Never promote a report-only lane to required from one green run. Require at least 14 days and 30 representative pull requests with at least 99% complete execution, then update the ruleset and this table together.

Change checklist

Update this file in the same pull request when any of these change:

  • workflow triggers, job names, timeouts, or nextest profile ownership;
  • required status contexts or strict/merge-queue policy;
  • scheduled cadence, alert routing, artifact contract, or local reproduction;
  • report-only versus gating semantics.

Do not copy per-module test counts here. Update e2e-suite-inventory.md and its enforced membership digest instead.