Object I/O (GET/PUT) tuning A/B matrix runbook

Scope: parameter tuning — measuring the effect of changing one RUSTFS_* runtime knob at a time, against a fixed binary. This is deliberately different from the code-change A/B gate in hotpath-warp-ab-runbook.md and the formal ABBA validation in hotpath-warp-abba-runbook.md, which compare a baseline binary against a candidate binary.

When a knob change turns out to need a code change, use those two runbooks for the code-level validation and come back here for the knob-level sweep.

1. What this runbook answers

For each tuning knob it answers three questions:

1. Which stage is actually slow — set_disk_encode, set_disk_rename, metadata_fanout, bitrot_verify, etc.?
2. Is the knob the real bottleneck, or is the stage slow for another reason?
3. Does widening/loosening the knob buy throughput without an unacceptable memory (RSS) or tail-latency regression?

The core discipline is one variable per A/B cell. Never change two knobs in the same cell, or the result is unexplainable.

2. Prerequisites

• Linux bench host (or an ansible-managed cluster); a laptop smoke run is too noisy to decide anything.
• warp on PATH (or pass --warp-bin to the driver).
• The observability metrics runtime enabled. The stage histograms below are not emitted when RUSTFS_OBS_METRICS_EXPORT_ENABLED=false or the runtime is otherwise off — see hotpath-warp-ab-runbook.md for the no-log / no-monitor baseline env.
• A warm, disposable data set. Recreate the bucket per run; do not bench against production data.

Load driver and gate are reused, not reimplemented:

• scripts/run_object_batch_bench_enhanced.sh — warp driver with rounds, median aggregation, baseline_compare.csv, and Prometheus service-metric capture.
• scripts/hotpath_warp_ab_gate.sh — relative budget gate over the deltas.
• scripts/run_hotpath_warp_ab.sh — optional orchestrator when a knob needs the full baseline-vs-candidate treatment (e.g. two different defaults).

3. Fixed test conditions (lock before you start)

Record these per run; a result without them is not reproducible:

nodes, disks_per_node, total_disks, cpu_per_node, mem_per_node,
network, erasure_set_drive_count, endpoint_mode (direct|lb),
rustfs_commit_sha, warp --version, durability mode

Workload matrix (the same shapes the hotpath gate uses, expanded for the stage-breakdown object sizes):

Concurrency ladder: 8, 16, 32, 64 (add 96, 128 on a bigger rig). Duration 120s, --rounds >= 3, cooldown >= 30s.

Isolate background noise before the sweep: scanner deep-verify, heal, replication, lifecycle transition, periodic capacity refresh — record whether each is on rather than silently assuming it is off.

4. Measurement stack

The code already instruments every stage below. Drive each A/B cell with these histograms (names verified against crates/io-metrics/src/lib.rs):

• PUT stages: rustfs_s3_put_object_stage_duration_ms{stage=...} — compute P50/P95/P99 per stage. Stages: app_bucket_validate, app_sse_config_lookup, app_object_lock_config_lookup, app_put_opts_build, app_prelookup, ingress_prepare, app_encryption_prepare, app_replication_decision, app_store_put, app_post_store_bookkeeping, app_capacity_update, set_disk_writer_setup, set_disk_encode, set_disk_rename, set_disk_old_data_cleanup.
• GET stages: rustfs_io_get_object_stage_duration_seconds{path=..., stage=...} — the path label separates the read paths: legacy_duplex, codec_streaming, direct_memory, body_cache, inline_direct, internal_meta, remote_transition, set_disk, empty. Stages: metadata, metadata_cache_lookup, metadata_fanout, metadata_resolve, object_info, path_decision, quorum_reached, range, reader_setup, stripe_read, stripe_read_first_shard, stripe_read_quorum, decode, reconstruct, emit, fill, output_poll, output_lock_wait, bitrot_verify, first_byte, full_body, response_handoff, lock_acquire.
• EC memory pressure: rustfs_ec_encode_inflight_bytes_current and the allocator reclaim gauge; plus node RSS and CPU.

Host telemetry (collect alongside every cell):

pidstat -durh 5 > telemetry/pidstat.txt &
mpstat 5 > telemetry/mpstat.txt &
iostat -xz 5 > telemetry/iostat.txt &

5. Tuning knob catalog

Defaults are verified against crates/config/src/constants/object.rs and crates/ecstore/src/erasure/coding/encode.rs.

5.1 PUT

5.2 GET

6. A/B matrix

Run each row as an independent cell. Baseline is the shipped default; candidate is one knob moved. Everything else (topology, sizes, concurrency, rounds, durability) stays fixed.

6.1 PUT

6.2 GET

7. Execution sequence

1. Freeze the conditions in §3 and record the provenance block.
2. Run the baseline cell (all defaults) and capture stage histograms + host telemetry.
3. Pick the one most-likely knob from the analysis. For large-object PUT that is almost always set_disk_encode → P1; for GET it is G1 (the legacy_duplex → codec_streaming switch).
4. Sweep that knob's candidate column one value at a time, same workload.
5. Read the decision table in §8; if the stage did not move, the knob is not the bottleneck — stop widening it and pick the next stage.

Driver invocation for one PUT cell:

scripts/run_object_batch_bench_enhanced.sh \
  --tool warp --endpoint http://127.0.0.1:9000 \
  --access-key "$RUSTFS_ACCESS_KEY" --secret-key "$RUSTFS_SECRET_KEY" \
  --bucket rustfs-put-tuning --warp-mode put \
  --sizes 16MiB,32MiB --concurrency 32 --duration 120s --rounds 3 \
  --out-dir target/bench/put-tuning-p1-64mib

8. Interpretation / decision table

9. Guardrails

• Never weaken correctness for throughput: read/write quorum, bitrot verify, xl.meta validation, and durability (RUSTFS_DURABILITY_MODE) are integrity contracts, not knobs. P5 and G6 are ceiling measurements and must be labelled as such; do not carry their values into production without an explicit durability/correctness decision.
• One variable per cell. A cell that changes two knobs is thrown away.
• Memory is part of the result. A throughput win with unbounded RSS growth is a regression; record RSS and the EC in-flight gauge for every PUT cell.
• Startup-latched knobs (RUSTFS_GET_SEEK_BUFFER_ENABLE, RUSTFS_GET_OUTPUT_HANDOFF_ATTRIBUTION_ENABLE, and the codec-streaming switches) require a process restart to change — see get-path-experimental-switches.md.
• Archive the raw data. Keep the baseline_compare.csv, median_summary.csv, stage histograms, and host telemetry per cell; the conclusion must trace back to them. Do not commit benchmark result snapshots to the repo — record them in the issue tracker.