Scanner Runtime Controls

This document describes the runtime controls and status fields for the RustFS data scanner. It is written for operators who need to reduce scanner pressure, diagnose slow scan progress, or confirm that background lifecycle, replication, heal, bitrot, and usage work is still moving.

For reproducible scanner-pressure validation and before/after evidence, see Scanner Benchmark Runbook.

What the scanner does

The scanner is the background maintenance loop that walks stored objects and feeds several subsystems:

• usage accounting and data usage cache updates;
• lifecycle expiry and transition admission;
• bucket replication repair admission;
• scanner-originated heal and bitrot checks;
• namespace alerts for excessive versions, retained version size, and folder fan-out.

Slowing the scanner can reduce idle CPU and disk pressure, but it also delays the maintenance work above. Prefer using the status fields below before changing cycle or pacing values.

Configuration Sources

Scanner runtime config is resolved in this order:

1. Environment variables.
2. Persisted admin config for the scanner subsystem.
3. Built-in defaults or speed preset-derived values.

Bitrot cycle resolution is slightly different because the canonical persistent key belongs to the heal subsystem:

1. RUSTFS_SCANNER_BITROT_CYCLE_SECS.
2. heal.bitrot_cycle.
3. Legacy compatibility key scanner.bitrot_cycle.
4. Built-in default.

The /v3/scanner/status response reports each effective runtime value with a source of env, config, scanner_compat_config, or default.

Runtime Controls

The fastest, fast, default, slow, and slowest presets set the base sleep multiplier, maximum wait, and cycle interval. Use scanner.delay, scanner.max_wait, and scanner.cycle when the preset is close but one axis needs a precise override.

Single-disk clean-idle scheduling

An erasure single-disk deployment using the built-in cycle and bitrot defaults automatically backs off repeated clean idle scans instead of walking the same unchanged namespace every minute. Each successful timer-driven cycle that finds no dirty usage or unresolved maintenance work doubles the next interval. The status endpoint reports the effective interval and multiplier.

The backoff is reset to the base interval by object or bucket mutations, lifecycle or replication configuration changes, partial or failed cycles, usage persistence failures, and unresolved scanner-originated heal or bitrot work. Active lifecycle or replication rules keep the base cadence. An explicit cycle, a non-default persisted speed, any environment speed or start-delay override, an environment bitrot override, or a non-default persisted active bitrot cycle also keeps the configured cadence rather than applying the automatic policy. Persisting scanner.speed=default or the default bitrot cycle is normalized to the built-in default and therefore keeps automatic scheduling enabled.

Lifecycle and replication configuration inspection is bounded so a slow metadata read cannot stall scanner startup or scheduling. A failed or timed-out inspection keeps the base cadence and is retried after 5 minutes, doubling up to a maximum of 60 minutes while failures continue. A lifecycle or replication configuration change wakes the scanner and retries inspection immediately.

With the default 30-day bitrot cycle, the clean-idle interval is capped at the bitrot cycle divided by the object selection window. With the default selection window this is about 42 minutes, which preserves the intended wall-clock bitrot coverage. If periodic bitrot is disabled, the clean-idle policy cap is 24 hours. The effective interval is jittered by up to 10 percent to avoid synchronized scanner starts.

Status Endpoint

The scanner status route is:

GET /v3/scanner/status

The request must be authenticated with an admin identity that has ServerInfoAdminAction. The JSON response has three scanner-specific top-level objects:

• runtime_config: the effective runtime controls and their value sources.
• cycle_schedule: the current effective cycle interval and clean-idle backoff state.
• metrics: scanner work, pressure, checkpoint, lifecycle, replication, heal, bitrot, and alert counters.

Example fields to inspect:

runtime_config.speed.value
runtime_config.delay.value
runtime_config.max_wait_seconds.value
runtime_config.cycle_interval_seconds.value
runtime_config.bitrot_cycle_seconds.value
cycle_schedule.effective_interval_seconds
cycle_schedule.clean_idle_backoff_enabled
cycle_schedule.clean_idle_backoff_multiplier
metrics.pacing_pressure.primary_pressure
metrics.pacing_pressure.last_cycle_budget_limited
metrics.lifecycle_transition.current_queued
metrics.lifecycle_transition.scanner_missed
metrics.maintenance_control.primary_control
metrics.source_work
metrics.replication_repair
metrics.scan_checkpoint

Reading Pacing Pressure

metrics.pacing_pressure.primary_pressure summarizes the highest-priority scanner pressure signal:

The ratio fields are fractions of the last cycle duration:

• last_cycle_throttle_sleep_ratio
• last_cycle_yield_ratio
• last_cycle_total_pause_ratio

If CPU is high but pause ratios are already high, increasing scanner.delay or scanner.max_wait may have limited value. Check active paths, source work, and disk activity before changing the cycle interval.

Reading Source Work

metrics.source_work, metrics.current_cycle_source_work, and metrics.last_cycle_source_work group scanner work by source:

• usage
• lifecycle
• bucket_replication
• site_replication
• heal
• bitrot
• alerts

Each source has checked, queued, executed, failed, skipped, and missed counters. missed means the scanner found work but could not admit it to the downstream queue. skipped means the work was intentionally merged or deduplicated.

Use these counters to decide whether scan progress is limited by scanner pacing or by a downstream subsystem such as lifecycle transition, replication repair, or heal admission.

Reading Heal Operations

The background heal status route is:

POST /v3/background-heal/status

It reports scanner-driven bitrot state together with heal queue execution state. healQueueLength and healActiveTasks keep the legacy totals. healOperations adds the same totals split by request source and priority:

Use this route when metrics.source_work shows heal or bitrot queued or missed work. Scanner-originated object checks should appear under scanner/low for opportunistic work, while manual admin heal should appear under admin/high. If scanner work grows but admin work remains blocked, treat that as heal queue pressure rather than scanner pacing pressure.

Reading Replication Repair

metrics.replication_repair, metrics.current_cycle_replication_repair, and metrics.last_cycle_replication_repair split scanner-discovered replication repair work by source and repair kind.

Each entry has the same checked, queued, executed, failed, skipped, and missed counters used by source_work, plus:

For bucket replication, queued means scanner-discovered repair was admitted to the replication queue, missed means the queue or worker path could not accept it, and skipped means the object did not require a new repair task.

The site replication kinds keep passive scanner discovery separate from active resync. Scanner status may report site replication boundary counters, but the scanner should not be treated as the active site replication resync controller.

Use this boundary when interpreting replication pressure:

If site_replication counters grow while bucket replication counters stay flat, investigate site replication status and resync state before tuning scanner pacing. If bucket_replication missed grows, investigate the bucket replication worker queue or target health before changing scanner cycle settings.

Reading Maintenance Control

metrics.maintenance_control derives a source-level control snapshot from scanner pacing, partial-cycle state, source work, and lifecycle transition queue state. It does not change scanner scheduling by itself; it explains why a source is moving, deferred, or blocked. When no scan cycle is currently active, source-work controls use the last completed cycle so recently missed work stays visible between scanner passes.

metrics.maintenance_control.primary_control summarizes the highest-priority source state:

Each metrics.maintenance_control.sources[] entry has:

Use this snapshot before changing scanner controls. For example, blocked_source with lifecycle/missed_work points at downstream lifecycle admission, while deferred_source with usage/partial_cycle points at scanner cycle budgets. lifecycle/expiry_queue_backlog means scanner-driven expiry or delete work is still queued or active in the expiry worker pool. lifecycle/transition_failed means transition worker execution failed during the current or last completed scan cycle, while lifecycle/transition_compensation_backlog means transition compensation is still pending or running after queue backpressure.

metrics.lifecycle_expiry exposes the expiry/delete worker queue observed by scanner-driven lifecycle work:

Reading Distributed Metrics

/rustfs/admin/v3/scanner/status and /rustfs/admin/v3/metrics report the node that handles the HTTP request. The metrics endpoint does not fan out to peer nodes. In distributed deployments, query every node explicitly and keep by-host=true enabled so each response includes that node's host view:

for endpoint in http://node-a:9000 http://node-b:9000 http://node-c:9000; do
  node="${endpoint#http://}"
  node="${node%%:*}"
  awscurl \
    --service s3 \
    --region us-east-1 \
    --access_key "$RUSTFS_ACCESS_KEY" \
    --secret_key "$RUSTFS_SECRET_KEY" \
    --request GET \
    "${endpoint}/rustfs/admin/v3/metrics?types=1&by-host=true&n=1" \
    > "artifacts/scanner-metrics.${node}.$(date -u +%Y%m%dT%H%M%SZ).ndjson"
done

The aggregated.scanner payload preserves the same scanner progress, checkpoint, pacing, source work, maintenance control, lifecycle expiry, and lifecycle transition fields used by the local scanner status, but only for the node that returned the response. The by_host.*.scanner payload keeps that node's host view. Compare the per-node artifacts externally to find old active paths, partial checkpoints, pacing pressure, source-level control pressure, or downstream queue admission problems across the deployment.

Reading Lifecycle Transition Status

metrics.lifecycle_transition focuses on scanner-driven lifecycle transition work:

When scanner_missed or queue_full rises, scanner lifecycle work is finding transition candidates faster than the transition queue can accept them. That is a downstream transition pressure signal, not just a scanner walk pressure signal.

Tuning Workflow

For symptoms where a mostly idle single-node, single-disk deployment has sustained CPU usage while the scanner is enabled:

1. Read /v3/scanner/status.
2. Check metrics.pacing_pressure.primary_pressure.
3. Check metrics.maintenance_control.primary_control and source entries before changing runtime controls.
4. Check runtime_config.delay, runtime_config.max_wait_seconds, and runtime_config.cycle_interval_seconds to confirm the active values and their sources.
5. Check metrics.current_cycle_objects_scanned, metrics.current_cycle_directories_scanned, and active paths to confirm the scanner is the active work.
6. If primary_pressure is throttle_pause and pause ratios are low, raise scanner.delay first.
7. If individual sleeps are too short, raise scanner.max_wait.
8. If each scan cycle finishes but starts too often, raise scanner.cycle.
9. If scans must be broken into bounded chunks, set one of the cycle budgets: scanner.cycle_max_duration, scanner.cycle_max_objects, or scanner.cycle_max_directories.
10. Recheck pacing_pressure, maintenance_control, source work, and lifecycle transition status after one or more scanner cycles.

Do not rely only on a longer cycle interval if lifecycle, replication, heal, or bitrot work must keep moving. Use source work and transition status to confirm that background maintenance is still making progress.

Helm

The Helm chart exposes the scanner environment variables under config.rustfs.scanner. Example:

config:
  rustfs:
    scanner:
      speed: "slow"
      delay: "30"
      max_wait_secs: "15"
      cycle_secs: "3600"
      cycle_max_duration_secs: "1800"
      cycle_max_objects: "1000000"
      cycle_max_directories: "100000"
      idle_mode: "true"
      yield_every_n_objects: "128"
      bitrot_cycle_secs: "2592000"

Use extraEnv for experimental or unrelated environment variables that are not represented by chart values.