docs/en/faq/operation_maintenance_faq.md
This topic provides answers to some questions related to operation and maintenance.
trash directory be cleaned up?You can configure the FE parameter catalog_trash_expire_second to specify how long files stay in the FE trash directory (Default: 24 hours).
The BE parameter trash_file_expire_time_sec controls BE trash cleanup intervals (Default: 24 hours).
After a DROP TABLE or DROP DATABASE, the data first enters the FE trash and is kept for one day, during which it can be recovered with RECOVER. After that, it moves to the BE trash, which also keeps it for 24 hours.
If the table property replicated_storage is set to true, writes use a primary–secondary mechanism: data is written to the primary replica first and then synchronized to others. Multiple replicas generally do not have a major impact on query performance.
You can check the cpucostns and memcostbytes fields in fe.audit.log.
The error indicates that the aggregation type of the materialized view must match that of the base table. For Unique Key tables, you can only materialized views to adjust their sort key order.
For example, if base table tableA has columns k1, k2, k3, with k1 and k2 as sort keys. While your queries contain the clause WHERE k3=x and need to be accelerated prefix index, you may create a materialized view that uses k3 as the first column:
CREATE MATERIALIZED VIEW k3_as_key AS
SELECT k3, k2, k1
FROM tableA;
query_latency the metric for average response time?Table-level import metrics can be obtained from: http://user:password@fe_host:http_port/metrics?type=json&with_table_metrics=all.
Cluster-level data can be retrieved from: http://fe_host:http_port/api/show_data (increment must be calculated manually).
query_latency provides percentile query response time.
SHOW PROC '/backends' retrieve metadata from the current FE and may lag. While SHOW BACKENDS retrieve metadata from the Leader FE and is authoritative.
Yes. You can configure the system variable wait_timeout (Default: 8 hours) to adjust the connection timeout.
Example:
SET GLOBAL wait_timeout = 3600;
No. Statements like GRANT <priv> on db1.tb1, db1.tb2 are not supported.
Subset revocation is not supported. It is recommended to grant privileges at the database or table level.
Table-level access control is supported. Row- and column-level access controls are not supported in the Open-source Edition.
No. Hot–cold separation is based on partitions.
Yes. Update storage_root_path in be.conf to add a new disk, and use semicolons to separate paths.
Run SHOW FRONTENDS; and check the Version field.
From v2.3 onwards, Primary Key tables supports full DELETE WHERE syntax. See Reference - DELETE for details.
No. Set it to a very large value.
Decommission the BE. After repair, add it back to the cluster.
No. Default is HDD. Manual configuration is required.
Yes. You do not need to wait until the rebalancing completed. Up to two nodes can be decommissioned at once.
Rebalancing happens automatically and should not affect normal operations. It is recommended to remove nodes one at a time.
Add six new BE nodes, and then decommission the old ones one by one.
Check for single-replica tables. Other repairs may be blocked if they continuously retry.
A large memory allocation request occurred, often caused by large queries. Check the corresponding query_id in be.INFO to locate the SQL.
Yes, temporary I/O fluctuation is expected during balancing.
distcp to migrate the data files, and then use Broker Load to load the data into the target cluster.The disk is full. Scale up the storage or clean trash.
It is likely because the storage_root_path is HDD and the table property storage_medium is set to SSD. You can set the table property to HDD:
ALTER TABLE db.table MODIFY PARTITION (*) SET("storage_medium"="HDD");
No. Use the recommended method: add a new node, and then remove the old one.
Balancing assumes equal-sized disks. Use disks of the same size to ensure even distribution.
max_backend_down_time_second is set to 3600, does it mean that I must recover the failed BE within one hour?If the downtime of the BE exceeds this configuration, FE will replenish replicas on other BE. When the failed BE is added back to the cluster, large rebalancing costs may occur.
Yes. See olapdb-tool.
Yes.
Example:
SELECT /*+ SET_VAR(query_timeout=1, is_report_success=true, parallel_fragment_exec_instance_num=2) */ COUNT(1)
FROM table;
VARCHAR columns are truncated based on actual length. Only the first column gets a short-key index. Place VARCHAR sort keys in the third position if possible.
BE process is still running. Kill the daemon process and restart.
No. Write requests automatically route to the leader FE; observers serve read-only queries.
LOG_FILE_NOT_FOUND caused by too many open files. How can I solve this?Check OS file descriptor limits by running cat /proc/$pid/limits, and run lsof -n -p $fe_pid>/tmp/fe_fd.txt to inspect the file descriptor in use.
4096 (configurable via the FE configuration max_partitions_in_one_batch).Stop the current Leader, and a new leader will be elected automatically.
Dynamic partition check runs every 10 minutes. This behavior is controlled by the FE configuration dynamic_partition_check_interval_seconds.
It is likely because client-side disconnects, connection pool drops, or network issues. Check OS backlog metrics and network stability.
Check if the following metrics have changed:
netstat -s | grep -i LISTEN
netstat -s | grep TCPBacklogDrop
cat /proc/sys/net/core/somaxconn
Immediately.
Run the following command:
SHOW TABLET FROM db.table PARTITION (<partition_name>);
It is an FE–BE communication failure. Check IP and port connectivity.
FQDN-mode deployment is recommended.
No. You can create a new partitioned table and use INSERT INTO SELECT to migrate the data.
Yes. See fe.audit.log.
Run the following SQL:
ALTER TABLE db.tbl MODIFY PARTITION (*) SET ("storage_medium"="SSD");
information_schema.COLUMNS show delay. Is this normal?Yes. ALTER operations are asynchronous. Check progress with SHOW ALTER COLUMN.
Follow these steps:
Disable dynamic partitions.
ALTER TABLE db.tbl SET ("dynamic_partition.enable" = "false");
Manually add partitions.
ALTER TABLE db.tbl ADD PARTITIONS START ("2019-01-01") END ("2019-12-31") EVERY (interval 1 day);
Re-enable dynamic partitions.
ALTER TABLE db.tbl SET ("dynamic_partition.enable" = "true");
Yes. StarRocks supports monitoring metrics for Routine Load tasks and can be connected to alert systems.
Run the following statements to identify UnhealthyTablets:
SHOW PROC '/statistic'
SHOW PROC '/statistic/<db_id>'
Then, analyze UnhealthyTablets with SHOW TABLET tablet_id.
If the result shows that two replicas have consistent data but one replica has inconsistent data—meaning two out of three replicas completed the write successfully—this is considered a successful write. You can then check whether the tablets in UnhealthyTablets are fixed. If they are fixed, it indicates an issue. If the status is changing, you can adjust the loading frequency for the corresponding table.
Increase per-user limits by running the following command:
ALTER USER 'jack' SET PROPERTIES ('max_user_connections'='1000');
Also check load balancers and idle connection accumulation (wait_timeout).
StarRocks typically performs better with XFS.
If BE recovers later, its replicas are deleted.
Tune the following FE configurations:
ADMIN SET FRONTEND CONFIG ("schedule_slot_num_per_path"="8");
ADMIN SET FRONTEND CONFIG ("max_scheduling_tablets"="1000");
ADMIN SET FRONTEND CONFIG ("max_balancing_tablets"="1000");
BACKUP operations are parallel, but upload to HDFS uses a single worker, which is controlled by the BE configuration upload_worker_count. Adjust it with caution because it might affect the disk I/O and network I/O.
Increase FE JVM memory.
apache_hdfs_broker.log for error "receive a check path request, request detail", and identify the duplicate files.Run SHOW PROC '/statistic' and ensure UnhealthyTablet count is zero.
Run the following command:
ALTER TABLE db.tbl MODIFY PARTITION (*) SET("replication_num"="3");
Yes, if enough BE nodes are available.
Check load balancers (ProxySQL, F5), idle connection accumulation, and reduce wait_timeout to 2–4 hours.
Metadata resides in FE. With only one FE, it's unrecoverable. With multiple FEs, you can re-add the failed node and metadata will be replicated.
Add JVM option -XX:-OmitStackTraceInFastThrow and restart FE to get full stack trace.
The partition columns must be key columns.