docs/en/administration/management/monitoring/metric_details/s.md
schema_change_mem_bytessegment_flush_queue_countsegment_metadata_mem_bytessegment_readsegment_replicate_queue_countsegment_zonemap_mem_bytesshort_key_index_mem_bytessmall_file_cache_countsnmp/proc/net/snmp.starrocks_be_clone_task_copy_bytesstarrocks_be_clone_task_copy_duration_msstarrocks_be_exec_state_report_active_threadsstarrocks_be_exec_state_report_queue_countstarrocks_be_exec_state_report_running_threadsstarrocks_be_exec_state_report_threadpool_sizestarrocks_be_files_scan_num_bytes_readfile_format, scan_type.starrocks_be_files_scan_num_files_readfile_format, scan_type.starrocks_be_files_scan_num_raw_rows_readfile_format, scan_type.starrocks_be_files_scan_num_rows_returnfile_format, scan_type.starrocks_be_files_scan_num_valid_rows_readfile_format, scan_type.starrocks_be_mem_pool_mem_limit_bytesstarrocks_be_mem_pool_mem_usage_bytesstarrocks_be_mem_pool_mem_usage_ratiostarrocks_be_mem_pool_workgroup_countstarrocks_be_pipe_prepare_pool_queue_lenstarrocks_be_priority_exec_state_report_active_threadsstarrocks_be_priority_exec_state_report_queue_countstarrocks_be_priority_exec_state_report_running_threadsstarrocks_be_priority_exec_state_report_threadpool_sizestarrocks_be_resource_group_cpu_limit_ratiostarrocks_be_resource_group_cpu_use_ratiostarrocks_be_resource_group_mem_inuse_bytesstarrocks_be_resource_group_mem_limit_bytesstarrocks_be_segment_file_not_found_totalstarrocks_be_staros_shard_info_fallback_totalg_starlet->get_shard_info()) that the BE's StarOSWorker had to issue because the requested shard info was not in the local cache (i.e. the FE had not pushed the shard to this BE before a query/compaction/lake operation referenced it). Only counted when the starlet readiness check passes and the RPC is actually dispatched; starlet-not-ready timeouts are not included. Should normally be near zero. A sustained or rising rate is a strong signal that FE-side task or node selection is scheduling work on a BE that does not yet have the shard, or that shard push propagation from FE is lagging. Recommended alert: high per-BE rate over a 5-minute window.starrocks_be_staros_shard_info_fallback_failed_totalstarrocks_be_staros_shard_info_fallback_total where the starmgr RPC returned a non-OK status. Use the ratio failed_total / fallback_total to alert on transient starmgr errors separately from routine successful fallbacks.starrocks_be_staros_shard_countStarOSWorker::add_shard and StarOSWorker::remove_shard (push-on-mutation), so the value reflects the last shard table mutation rather than being recomputed at scrape time. The gauge is not reset on BE shutdown and will retain its last value until the next mutation. Use it to observe shard distribution balance across BEs and to detect drift from the FE-side placement.starrocks_fe_clone_task_copy_bytesstarrocks_fe_clone_task_copy_duration_msstarrocks_fe_clone_task_successstarrocks_fe_clone_task_totalstarrocks_fe_last_finished_job_timestampstarrocks_fe_memory_usagestarrocks_fe_meta_log_count100000 is considered reasonable..starrocks_fe_publish_version_daemon_loop_totalpublish-version-daemon loop runs on this FE node.The following metrics are summary-type metrics that provide latency distributions for different phases of a transaction. These metrics are reported exclusively by the Leader FE node.
Each metric includes the following outputs:
quantile label, which can have values of 0.75, 0.95, 0.98, 0.99, and 0.999.<metric_name>_sum: The total cumulative time spent in this phase, for example, starrocks_fe_txn_total_latency_ms_sum.<metric_name>_count: The total number of transactions recorded for this phase, for example, starrocks_fe_txn_total_latency_ms_count.All transaction metrics share the following labels:
type: Categorizes transactions by their load job source type (for example, all, stream_load, routine_load). This allows for monitoring both overall transaction performance and the performance of specific load types. The reported groups can be configured via the FE parameter txn_latency_metric_report_groups.is_leader: Indicates whether the reporting FE node is the Leader. Only the Leader FE (is_leader="true") reports actual metric values. Followers will have is_leader="false" and report no data.starrocks_fe_query_resource_groupstarrocks_fe_query_resource_groupstarrocks_fe_query_resource_group_errstarrocks_fe_query_resource_group_errstarrocks_fe_query_resource_group_latencystarrocks_fe_query_resource_group_latencystarrocks_fe_routine_load_error_rowsstarrocks_fe_routine_load_jobsUnit: Count
Description: The total number of Routine Load jobs in different states. For example:
starrocks_fe_routine_load_jobs{state="NEED_SCHEDULE"} 0
starrocks_fe_routine_load_jobs{state="RUNNING"} 1
starrocks_fe_routine_load_jobs{state="PAUSED"} 0
starrocks_fe_routine_load_jobs{state="STOPPED"} 0
starrocks_fe_routine_load_jobs{state="CANCELLED"} 1
starrocks_fe_routine_load_jobs{state="UNSTABLE"} 0
starrocks_fe_routine_load_max_lag_of_partitionenable_routine_load_lag_metrics is set to true and the offset lag is greater than or equal to the FE configuration min_routine_load_lag_for_metrics. By default, enable_routine_load_lag_metrics is false, and min_routine_load_lag_for_metrics is 10000.starrocks_fe_routine_load_max_lag_time_of_partitionenable_routine_load_lag_time_metrics is set to true. By default, enable_routine_load_lag_time_metrics is false.starrocks_fe_routine_load_pausedstarrocks_fe_routine_load_receive_bytesstarrocks_fe_routine_load_rowsstarrocks_fe_safe_mode0 (disabled) and 1 (enabled). When Safe Mode is enabled, the cluster no longer accepts any loading requests.starrocks_fe_scheduled_pending_tablet_numstarrocks_fe_scheduled_running_tablet_numstarrocks_fe_slow_lock_held_time_msslow_lock_threshold_ms configuration parameter. It tracks the maximum lock held time among all lock owners when a slow lock event is detected. Each metric includes quantile values (0.75, 0.95, 0.98, 0.99, 0.999), _sum, and _count outputs. Note: This metric may not accurately reflect the exact lock held time under high contention, because the metric is updated once the wait time exceeds the threshold, but the held time may continue to increase until the owner completes its operation and releases the lock. However, this metric can still be updated even when deadlock occurs.starrocks_fe_slow_lock_wait_time_msslow_lock_threshold_ms configuration parameter. It accurately tracks how long threads wait to acquire locks during lock contention scenarios. Each metric includes quantile values (0.75, 0.95, 0.98, 0.99, 0.999), _sum, and _count outputs. This metric provides precise wait time measurements. Note: This metric cannot be updated when deadlock occurs, hence it cannot be used to detect deadlock situations.starrocks_fe_sql_block_hit_countstarrocks_fe_tablet_pre_split_eligibility_skippedreason — the SkipReason enum value (lower-cased). Per-load values: not_range_distribution, table_not_normal, has_materialized_view_or_rollup, unsupported_sort_key, metadata_not_resolved, multiple_base_index_tablets, partition_not_empty, disabled_by_config, disabled_by_session. Multi-partition (P2-a) per-partition values: unsupported_partition_column_type (partition source column type cannot be projected, e.g. STRUCT/ARRAY), invalid_partition_value (sampled partition cell can't be formatted into an AddPartitionClause, e.g. null in a non-nullable column or unparseable date), grouper_empty (every sample row was dropped by the formatter/analyzer), stale_catalog_state (partition was seen by the grouper but disappeared before the coordinator re-resolved it under READ lock — concurrent partition drop/replace), partition_not_eligible_post_create (the post-pre-create eligibility re-check failed, typically because the partition is non-empty or now has multiple tablets).starrocks_fe_tablet_pre_split_sampler_invocationsstarrocks_fe_tablet_pre_split_sampler_failedreason — the post-eligibility failure category (lower-cased SkipReason), one of sample_failed (sampler executor threw), timeout_pre_submit (sample + plan + build phase exceeded tablet_pre_split_pre_submit_timeout_seconds), submit_failed (TabletReshardJobMgr rejected admission), pre_create_failed (multi-partition path: LocalMetastore.addPartitions threw while pre-creating a target partition — that one partition is dropped from the combined submit and falls back to BE runtime auto-create; sibling partitions in the same load continue).tablet_pre_split_eligibility_skipped (sampler never ran) and from tablet_pre_split_tier_used (which records the tier that succeeded). Meta-tier → data-tier fallback alone is not a failure; it is tracked via tablet_pre_split_tier_used{tier=data_tier}.starrocks_fe_tablet_pre_split_tier_usedtier — meta_tier (boundaries computed from Parquet/ORC row-group statistics; no row data read) or data_tier (boundaries computed from actual row samples collected via a FILES sub-query — covers both direct data-tier invocations and meta-tier → data-tier fallbacks).starrocks_fe_tablet_pre_split_boundaries_plannedeffectiveTabletCount - 1 (a K-tablet split needs K-1 cut points).starrocks_fe_tablet_pre_split_partitions_totalPartitionSamples entry that survived the grouper. Combined with tablet_pre_split_partitions_capped and the tablet_pre_split_pre_create{result=...} family this tells operators how many partitions each multi-partition invocation actually acts on. Stays at zero for the single-partition path.starrocks_fe_tablet_pre_split_partitions_cappedtablet_pre_split_max_partitions_per_load. The grouper keeps the partitions with the highest sample counts and drops the lowest-count tail; dropped partitions fall back to BE runtime auto-create with no pre-split. Sustained non-zero values mean the cap is biting — consider raising tablet_pre_split_max_partitions_per_load or reducing partition cardinality on the load.starrocks_fe_tablet_pre_split_pre_createresult — succeeded (LocalMetastore.addPartitions returned normally — the partition was created or silently deduped), failed (addPartitions threw, e.g. concurrent ALTER or journal failure; the affected partition falls back to BE runtime auto-create and is also recorded under tablet_pre_split_sampler_failed{reason=pre_create_failed}), already_exists (the partition was found in the catalog at pre-create time — concurrent loader race; the coordinator reuses the existing partition).LocalMetastore.addPartitions, broken down by outcome. Total attempts = sum of all three labels. Stays at zero for the single-partition path.starrocks_fe_tablet_pre_split_pre_submit_wait_mstablet_pre_split_pre_submit_timeout_seconds.starrocks_fe_tablet_pre_split_post_submit_wait_msFINISHED on the admitted Sample-Based Tablet Pre-Split reshard job. Fires on both production paths — the INSERT-from-FILES hook (called from StmtExecutor before StatementPlanner.plan opens the load txn) and the Broker Load hook (called from BrokerLoadJob.createLoadingTask before beginTxn opens T_load) — and on the optional runPreSplit synchronous-await wrapper used by tests. In all cases the trigger load itself plans against the post-split layout.starrocks_fe_tablet_pre_split_post_submit_hard_capFINISHED within tablet_pre_split_post_submit_wait_seconds. Fires on the INSERT-from-FILES production path on timeout (the INSERT then proceeds without abort against the currently visible tablet layout — still the original layout if the daemon hasn't transitioned, or partially / fully post-split if the daemon raced past the wait. tablet_pre_split_load_abort is NOT incremented because the INSERT itself is not aborted) and on the runPreSplit synchronous-await wrapper. The Broker Load production path does not await and so does not update this counter.starrocks_fe_tablet_pre_split_load_abortFINISHED in time. Sibling counter of tablet_pre_split_post_submit_hard_cap. Production load paths proceed without abort against the currently visible layout on post-submit timeout rather than abort, so this counter stays at zero in production today; it only fires when a caller uses the strict runPreSplit wrapper (tests, or a future caller that opts into abort-on-timeout).starrocks_fe_tablet_max_compaction_scorestarrocks_fe_tablet_numstarrocks_fe_txn_publish_ack_latency_msready-to-finish time to the final finish time when the transaction is marked as VISIBLE. This metric includes final acknowledgment steps after the transaction is ready to finish.starrocks_fe_txn_publish_can_finish_latency_mspublish task completion to the moment canTxnFinish() first returns true, measured from publish version finish time to ready-to-finish time.starrocks_fe_txn_publish_execute_latency_mspublish task, from when the task is picked up to when it finishes. This metric represents the actual time being spent to make the transaction's changes visible.starrocks_fe_txn_publish_latency_mspublish phase, from commit time to finish time. This is the duration it takes for a committed transaction to become visible to queries. It is the sum of the schedule, execute, can_finish, and ack sub-phases.starrocks_fe_txn_publish_schedule_latency_mscommit time to when the publish task is picked up. This metric reflects scheduling delays or queueing time in the publish pipeline.starrocks_fe_txn_total_latency_msprepare time to the finish time. This metric represents the full end-to-end duration of a transaction.starrocks_fe_txn_write_latency_mswrite phase of a transaction, from prepare time to commit time. This metric isolates the performance of the data writing and preparation stage before the transaction is ready to be published.starrocks_fe_unfinished_backup_job0.starrocks_fe_unfinished_querystarrocks_fe_unfinished_restore_job0.storage_page_cache_mem_bytesstream_loadstream_load_pipe_countstreaming_load_bytesstreaming_load_current_processingstreaming_load_duration_msstreaming_load_requests_total