Sui Validator Alert Reference - Sui

When running a Sui Validator node or Full node, you may want to configure alerting based off some or all of the following metrics.

Alert reference

The following sections cover the alert settings, but their details are meant to be customized in the following ways:

Replace $network with your actual network label (for example, mainnet, testnet, and so on).
Thresholds assume about 10,000 stake units — adjust for your own validator set size.
Labels like host and container are stripped to be agnostic on infrastructure.

High-priority chain health alerts (validator-specific)

These alerts should receive the most immediate attention from you or your team.

Safe mode during reconfiguration

Key	Value
Name	`Safe Mode during Reconfiguration`
Summary	Epoch failed to advance; chain entered safe mode
Duration	`5m`

is_safe_mode{network="$network"} > 0.5 or absent(is_safe_mode{network="$network"})

Consensus proposals failure

Key	Value
Name	`Consensus Proposals Failure`
Summary	Less than 80% of stake is proposing consensus blocks
Duration	`5m`

sum(
  sum by (host) (current_voting_right{network="$network"})
  and
  sum by (host) (rate(consensus_proposed_blocks{network="$network"}[5m])) > 0
) < 8000

Checkpoint execution rate is low

Key	Value
Name	`Checkpoint Execution Rate Is Low`
Summary	Less than 80% of stake is executing checkpoints quickly enough
Duration	`5m`

sum(
  sum by (host) (current_voting_right{network="$network"})
  and
  sum by (host) (rate(last_executed_checkpoint{network="$network"}[5m])) > 2
) < 8000

Certificate execution latencies are high

Key	Value
Name	`Certificate execution latencies are high`
Summary	Less than 80% of stake is handling shared-object tx certs with low enough latency
Duration	`5m`

sum(
  sum by (host) (current_voting_right{network="$network"})
  and
  histogram_quantile(0.95, sum by (le, host) (
    rate(validator_service_handle_certificate_consensus_latency_bucket{network="$network"}[5m])
  )) < 3
) < 8000

Randomness DKG failure

Key	Value
Name	`RandomnessDkgFailure`
Summary	Random beacon DKG has failed on one or more hosts
Duration	`5m`

epoch_random_beacon_dkg_failed{network="$network"} > 0 or absent(is_safe_mode{network="$network"})

Validators not upgraded

Key	Value
Name	`Mysten validators are not upgraded`
Summary	Validators are behind on protocol version
Duration	`1h`

min(sui_configured_max_protocol_version{network="$network", host=~"Mysten-.*"})
  < quantile(0.34, sui_configured_max_protocol_version{network="$network"})

⚠️ Non-urgent and warning alerts

All alerts are important, but the following alerts and warnings can be addressed within a normal node maintenance workflow.

Consensus sequencing p99 latency high

Key	Value
Name	`Consensus sequencing p99 latencies are high`
Summary	Less than 80% of stake is sequencing tx certs with acceptable latency
Duration	`1m`

sum(
  sum by (host) (current_voting_right{network="$network"})
  and
  histogram_quantile(0.95, sum by (le, host) (
    rate(sequencing_certificate_latency_bucket{network="$network", position="0", tx_type=~"shared_certificate|owned_certificate|soft_bundle"}[2m])
  )) < 2
) < 5000

System invariant violations

Key	Value
Name	`System Invariant Violations`
Summary	A system invariant violation was reported
Duration	`1m`

max(system_invariant_violations{network="$network"}) > 0