ClickHouse monitoring

ClickHouse server performance from built-in metrics data

SkyWalking leverages ClickHouse built-in metrics data since v20.1.2.4. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.

Data flow

1. Configure ClickHouse to expose metrics data for scraping from Prometheus.
2. OpenTelemetry Collector fetches metrics from ClickeHouse server through Prometheus endpoint, and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.

Set up

1. Set up built-in prometheus endpoint .
2. Set up OpenTelemetry Collector . For details on Prometheus Receiver in OpenTelemetry Collector, refer to here.
3. Config SkyWalking OpenTelemetry receiver.

ClickHouse Monitoring

ClickHouse monitoring provides monitoring of the metrics 、events and asynchronous_metrics of the ClickHouse server. ClickHouse cluster is cataloged as a Layer: CLICKHOUSE Service in OAP. Each ClickHouse server is cataloged as an Instance in OAP.

ClickHouse Instance Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
CpuUsage	count	meter_clickhouse_instance_cpu_usage	CPU time spent seen by OS per second(according to ClickHouse.system.dashboard.CPU Usage (cores)).	ClickHouse
MemoryUsage	percentage	meter_clickhouse_instance_memory_usage	Total amount of memory (bytes) allocated by the server/ total amount of OS memory.	ClickHouse
MemoryAvailable	percentage	meter_clickhouse_instance_memory_available	Total amount of memory (bytes) available for program / total amount of OS memory.	ClickHouse
Uptime	sec	meter_clickhouse_instance_uptime	The server uptime in seconds. It includes the time spent for server initialization before accepting connections.	ClickHouse
Version	string	meter_clickhouse_instance_version	Version of the server in a single integer number in base-1000.	ClickHouse
FileOpen	count	meter_clickhouse_instance_file_open	Number of files opened.	ClickHouse

ClickHouse Network Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
TcpConnections	count	meter_clickhouse_instance_tcp_connections
meter_clickhouse_tcp_connections	Number of connections to TCP server.	ClickHouse
MysqlConnections	count	meter_clickhouse_instance_mysql_connections
meter_clickhouse_mysql_connections	Number of client connections using MySQL protocol.	ClickHouse
HttpConnections	count	meter_clickhouse_instance_http_connections
meter_clickhouse_mysql_connections	Number of connections to HTTP server.	ClickHouse
InterserverConnections	count	meter_clickhouse_instance_interserver_connections
meter_clickhouse_interserver_connections	Number of connections from other replicas to fetch parts.	ClickHouse
PostgresqlConnections	count	meter_clickhouse_instance_postgresql_connections
meter_clickhouse_postgresql_connections	Number of client connections using PostgreSQL protocol.	ClickHouse
ReceiveBytes	bytes	meter_clickhouse_instance_network_receive_bytes
meter_clickhouse_network_receive_bytes	Total number of bytes received from network.	ClickHouse
SendBytes	bytes	meter_clickhouse_instance_network_send_bytes
meter_clickhouse_network_send_bytes	Total number of bytes send to network.	ClickHouse

ClickHouse Query Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
QueryCount	count	meter_clickhouse_instance_query
meter_clickhouse_query	Number of executing queries.	ClickHouse
SelectQueryCount	count	meter_clickhouse_instance_query_select
meter_clickhouse_query_select	Number of executing queries, but only for SELECT queries.	ClickHouse
InsertQueryCount	count	meter_clickhouse_instance_query_insert
meter_clickhouse_query_insert	Number of executing queries, but only for INSERT queries.	ClickHouse
SelectQueryRate	count/sec	meter_clickhouse_instance_query_select_rate
meter_clickhouse_query_select_rate	Number of SELECT queries per second.	ClickHouse
InsertQueryRate	count/sec	meter_clickhouse_instance_query_insert_rate
meter_clickhouse_query_insert_rate	Number of INSERT queries per second.	ClickHouse
Querytime	microsec	meter_clickhouse_instance_querytime_microseconds
meter_clickhouse_querytime_microseconds	Total time of all queries.	ClickHouse
SelectQuerytime	microsec	meter_clickhouse_instance_querytime_select_microseconds
meter_clickhouse_querytime_select_microseconds	Total time of SELECT queries.	ClickHouse
InsertQuerytime	microsec	meter_clickhouse_instance_querytime_insert_microseconds
meter_clickhouse_querytime_insert_microseconds	Total time of INSERT queries.	ClickHouse
OtherQuerytime	microsec	meter_clickhouse_instance_querytime_other_microseconds
meter_clickhouse_querytime_other_microseconds	Total time of queries that are not SELECT or INSERT.	ClickHouse
QuerySlowCount	count	meter_clickhouse_instance_query_slow
meter_clickhouse_query_slow	Number of reads from a file that were slow.	ClickHouse

ClickHouse Insertion Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
InsertQueryCount	count	meter_clickhouse_instance_query_insert
meter_clickhouse_query_insert	Number of executing queries, but only for INSERT queries.	ClickHouse
InsertedRowCount	count	meter_clickhouse_instance_inserted_rows
meter_clickhouse_inserted_rows	Number of rows INSERTed to all tables.	ClickHouse
InsertedBytes	bytes	meter_clickhouse_instance_inserted_bytes
meter_clickhouse_inserted_bytes	Number of bytes INSERTed to all tables.	ClickHouse
DelayedInsertCount	count	meter_clickhouse_instance_delayed_insert
meter_clickhouse_delayed_insert	Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition.	ClickHouse

ClickHouse Replicas Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
ReplicatedChecks	count	meter_clickhouse_instance_replicated_checks
meter_clickhouse_replicated_checks	Number of data parts checking for consistency.	ClickHouse
ReplicatedFetch	count	meter_clickhouse_instance_replicated_fetch
meter_clickhouse_replicated_fetch	Number of data parts being fetched from replica.	ClickHouse
ReplicatedSend	count	meter_clickhouse_instance_replicated_send
meter_clickhouse_replicated_send	Number of data parts being sent to replicas.	ClickHouse

ClickHouse MergeTree Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
BackgroundMergeCount	count	meter_clickhouse_instance_background_merge
meter_clickhouse_background_merge	Number of executing background merges.	ClickHouse
MergeRows	count	meter_clickhouse_instance_merge_rows
meter_clickhouse_merge_rows	Rows read for background merges. This is the number of rows before merge.	ClickHouse
MergeUncompressedBytes	bytes	meter_clickhouse_instance_merge_uncompressed_bytes
meter_clickhouse_merge_uncompressed_bytes	Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge.	ClickHouse
MoveCount	count	meter_clickhouse_instance_move
meter_clickhouse_move	Number of currently executing moves.	ClickHouse
PartsActive	Count	meter_clickhouse_instance_parts_active
meter_clickhouse_parts_active	Active data part, used by current and upcoming SELECTs.	ClickHouse
MutationsCount	count	meter_clickhouse_instance_mutations
meter_clickhouse_mutations	Number of mutations (ALTER DELETE/UPDATE).	ClickHouse

ClickHouse Kafka Table Engine Supported Metrics

When table engine works with Apache Kafka.

Kafka lets you:

• Publish or subscribe to data flows.
• Organize fault-tolerant storage.
• Process streams as they become available.

Monitoring Panel	Unit	Metric Name	Description	Data Source
KafkaMessagesRead	count	meter_clickhouse_instance_kafka_messages_read
meter_clickhouse_kafka_messages_read	Number of Kafka messages already processed by ClickHouse.	ClickHouse
KafkaWrites	count	meter_clickhouse_instance_kafka_writes
meter_clickhouse_kafka_writes	Number of writes (inserts) to Kafka tables.	ClickHouse
KafkaConsumers	count	meter_clickhouse_instance_kafka_consumers
meter_clickhouse_kafka_consumers	Number of active Kafka consumers.	ClickHouse
KafkaProducers	count	meter_clickhouse_instance_kafka_producers
meter_clickhouse_kafka_producers	Number of active Kafka producer created.	ClickHouse

ClickHouse ZooKeeper Supported Metrics

ClickHouse uses ZooKeeper for storing metadata of replicas when using replicated tables. If replicated tables are not used, this section of parameters can be omitted.

Monitoring Panel	Unit	Metric Name	Description	Data Source
ZookeeperSession	count	meter_clickhouse_instance_zookeeper_session
meter_clickhouse_zookeeper_session	Number of sessions (connections) to ZooKeeper.	ClickHouse
ZookeeperWatch	count	meter_clickhouse_instance_zookeeper_watch
meter_clickhouse_zookeeper_watch	Number of watches (event subscriptions) in ZooKeeper.	ClickHouse
ZookeeperBytesSent	bytes	meter_clickhouse_instance_zookeeper_bytes_sent
meter_clickhouse_zookeeper_bytes_sent	Number of bytes send over network while communicating with ZooKeeper.	ClickHouse
ZookeeperBytesReceive	bytes	meter_clickhouse_instance_zookeeper_bytes_received
meter_clickhouse_zookeeper_bytes_received	Number of bytes send over network while communicating with ZooKeeper.	ClickHouse

ClickHouse Keeper Supported Metrics

Monitoring Panel	Unit	Metric Name	Description	Data Source
KeeperAliveConnections	count	meter_clickhouse_instance_keeper_connections_alive
meter_clickhouse_keeper_connections_alive	Number of alive connections for embedded ClickHouse Keeper.	ClickHouse
KeeperOutstandingRequets	count	meter_clickhouse_instance_keeper_outstanding_requests
meter_clickhouse_keeper_outstanding_requests	Number of outstanding requests for embedded ClickHouse Keeper.	ClickHouse

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in /config/otel-rules/clickhouse. The ClickHouse dashboard panel configurations are found in /config/ui-initialized-templates/clickhouse.