docs/testing/README.md
Owner: backlog#1153 (infra-11). This file is the authoritative home for the test-layer taxonomy, naming conventions, and serial/nextest rules. The event × budget × required-status matrix is owned separately by
docs/testing/ci-gates.md(backlog#1149 ci-15); this file links to it rather than duplicating counts, timeouts, or required-check names.
RustFS layers its tests from cheap-and-narrow to expensive-and-broad. Higher layers catch what lower layers cannot but cost more wall-clock and setup, so each layer has a clear entry point and a clear "when". Pick the lowest layer that can prove your change; add a higher-layer test only when the behaviour is not observable below it.
| Layer | What it covers | Entry command | When it runs |
|---|---|---|---|
| Unit & crate integration | Per-crate logic and in-process integration tests, run under nextest | cargo nextest run --all --exclude e2e_test (or -p <crate>) | Every PR (required) |
| ecstore black-box | Erasure-coded read/write/recovery validation of the ecstore stack | scripts/run_ecstore_validation_suite.sh --profile quick | Local / release validation (not in CI workflows) |
e2e (e2e_test crate) | Full server spun up per test, driven over the S3 API | cargo nextest run --profile e2e-smoke -p e2e_test | PR smoke lane + scheduled full/nightly lanes |
| s3s-e2e conformance | External S3 conformance tool run against a live rustfs server | ./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data | Per-PR e2e gate (e2e-tests job) |
| S3 compatibility | Third-party suites: ceph/s3-tests (boto3) and MinIO mint (many SDKs) | scripts/s3-tests/run.sh (mint: .github/workflows/mint.yml) | s3-tests: per-PR gate; mint: scheduled, report-only |
| Chaos / fault-injection | Multi-node, power-loss, and disk-fault harness | — (harness planned) | Planned — tracked in backlog#1100 |
| Fuzz | cargo-fuzz targets over untrusted parsing/validation surfaces | ./scripts/fuzz/run.sh (or cd fuzz && cargo +nightly fuzz run <target>) | PR smoke + nightly corpus (.github/workflows/fuzz.yml) |
| Benchmarks | Criterion micro/throughput benchmarks | cargo bench -p <crate> | On-demand / local |
The When it runs column is a qualitative pointer only. The authoritative event × timeout × required-status matrix lives in
docs/testing/ci-gates.md(ci-15) — do not duplicate its numbers here.
Layer notes:
make test wraps
cargo nextest run --all --exclude e2e_test; CI runs the same set under the
strict ci profile (.config/nextest.toml). Requires cargo-nextest (see
Serial execution & nextest profiles).scripts/run_ecstore_validation_suite.sh has four
profiles (quick / full / destructive / fuzz); quick is the
PR-smoke-sized core read/write/recovery pass. It is a local and
release-validation tool, not wired into a CI workflow.e2e-smoke
profile is the single PR wiring mechanism; slow/cross-process suites run in
scheduled lanes (e2e-repl-nightly, and the reliability group). Full
contributor guide: crates/e2e_test/README.md;
per-module counts: e2e-suite-inventory.md.scripts/e2e-run.sh <binary> <data-dir>.ceph/s3-tests exercises S3 semantics through boto3
and is gated on a committed allow-list
(scripts/s3-tests/implemented_tests.txt); MinIO mint runs many real
client SDKs and is report-only until suites pass reliably (see the header of
.github/workflows/mint.yml).e2e-reliability group.)cargo-fuzz harness is an isolated sub-workspace under
fuzz/ (kept out of the root workspace on purpose); see
fuzz/README.md for targets and corpus rules.benches/. They
are not a gate; run them locally to compare before/after on a specific crate.Script inventory: every entry under scripts/ — including the runners above —
is indexed in scripts/README.md with its status
(ci-gate / dev-tool / archived) and wiring.
Fixed GHSA advisories map to named, discoverable regression tests. The
advisory -> test map lives in
docs/testing/security-regressions.md. sec-14
(backlog#1151) formalizes the written admission policy in AGENTS.md.
The migration-critical CI gate selects tests by name substring rather than by module path, so a rename that drops the substring silently thins the gate. These substrings are therefore reserved: keep them in the test name when a test proves migration-critical behaviour.
| Substring | Guards |
|---|---|
data_movement | Cross-pool / cross-set object data-movement proofs |
rebalance | Pool rebalance correctness |
decommission | Pool decommission correctness |
source_cleanup | Post-migration source cleanup |
delete_marker | Delete-marker handling across migration |
The gate is enforced by
scripts/check_migration_gate_count.sh,
which counts the selected tests and fails if the count drops below the committed
floor in
.config/migration-gate-floor.txt.
A deliberate reduction must lower the floor in the same PR, so the change is
reviewable in the diff (backlog#1153 infra-12). The substring list above is the
same one the gate uses; keep the two in sync when adding a reserved word.
..._regression_test, ..._issue_NNNN_...) or the invariant it protects, so
a reviewer can find the guard for a past bug by grepping..config/nextest.toml
(for example the _real_dual_node / _real_single_node markers route
replication tests into the nightly lane). Follow the existing marker when
adding a test to an established suite; see
crates/e2e_test/README.md.AGENTS.md).cargo-nextest is the runner. make test requires it and CI installs it;
plain cargo test is not a faithful substitute because the two runners execute
tests differently.
cargo install cargo-nextest --locked, or a prebuilt binary
(faster) from https://nexte.st/docs/installation/.RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test runs the
plain cargo test fallback, but the results are not authoritative —
serialization semantics differ from CI and [test-groups] do not apply.Why #[serial] is mostly a no-op under nextest. nextest runs every test in
its own process, so serial_test's in-process #[serial] mutex does not
serialize tests against each other. The mechanism that actually serializes
across nextest's process boundary is a nextest [test-groups] entry with
max-threads = 1 (for example ecstore-serial-flaky and e2e-reliability in
.config/nextest.toml). Consequences:
#[serial] expecting cross-test isolation under nextest. If two
tests genuinely share process/global state or a fixed external resource,
serialize them with a [test-groups] entry, or make each test self-isolating
(per-test instance context — see backlog#1153 infra-7 / infra-8).#[serial] markers only affect the
cargo test fallback. The serial-debt census and removal plan are tracked in
backlog#1153 infra-7.Profiles (all defined in .config/nextest.toml):
default — local runs. Never retries: a red test locally is a real
failure to investigate, not noise to retry away.ci — the strict CI gate: global retries = 0 plus a narrowly-scoped
quarantine list (retries = 2) for tests with a tracked OPEN flake issue.e2e-smoke — the PR smoke subset of the e2e_test crate (the single wiring
mechanism for e2e in PR CI).e2e-repl-nightly — the scheduled slow/cross-process replication lane.Time-driven tests should prefer paused time (tokio::time with start_paused
and advance) or explicit event synchronization over fixed sleep race
windows. The written convention and the docs/testing/time-control.md guide are
added by backlog#1153 infra-4.
Line coverage is measured weekly, not per-PR, and is non-blocking: it exists for visibility and trend, never as a required check. Per-crate ratchets for the security-critical crates (iam / kms / policy / crypto) build on this baseline later (backlog#1153 infra-6, report-only first).
.github/workflows/coverage.yml runs every Sunday and on manual
dispatch: cargo llvm-cov nextest --workspace --exclude e2e_test under the
ci nextest profile — the same scope and profile as the PR test gate. The
per-crate line-coverage table lands in the run's job summary; the lcov +
JSON exports are uploaded as a coverage-lcov-<run> artifact kept for
90 days. Scheduled failures open/append the [scheduled-failure] coverage
issue via the shared alert action (ci-8).make coverage is the equivalent (slow — instrumented rebuild
plus the full suite). It prints the same per-crate table via
scripts/coverage_per_crate.py and writes target/llvm-cov/lcov.info and
coverage.json.coverage-lcov-* artifacts and compare the lcov.info files with your lcov
tooling of choice.e2e_test crate (excluded from
the unit gate; its lanes are described in the taxonomy above).A flaky test is one that fails non-deterministically without a corresponding code change. Flakes erode trust in the gate and block tightening required checks, so they are handled on a strict, time-boxed loop.
Retry semantics (source of truth: .config/nextest.toml):
default profile never retries. A red test on your machine is
a real failure to investigate, not noise to paper over.ci profile runs with global retries = 0. A new race must fail
on its first occurrence so the first crime scene is never masked.retries = 2, and only under the
ci profile. Each quarantine entry MUST link exactly one OPEN issue.flaky in target/nextest/ci/junit.xml
(uploaded as a CI artifact). That marker — not a green check — is how we see
a flake is still live.Lifecycle of a flake:
flaky marker in the JUnit report..config/nextest.toml with a comment linking that OPEN issue. This grants
retries = 2 under CI so the flake stops reddening unrelated PRs, while the
flaky marker keeps it visible.First quarantine members: the two backlog#937 ecstore groups
(concurrent_resend_same_part_commits_one_generation and
store::bucket::tests::bucket_delete_*).