BookKeeper monitoring

SkyWalking leverages OpenTelemetry Collector to collect metrics data from the BookKeeper and leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System. Kafka entity as a Service in OAP and on the `Layer: BOOKKEEPER.

Data flow

1. BookKeeper exposes metrics through Prometheus endpoint.
2. OpenTelemetry Collector fetches metrics from BookKeeper cluster via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.`

Setup

1. Set up BookKeeper Cluster.
2. Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
3. Config SkyWalking OpenTelemetry receiver.

BookKeeper Monitoring

Bookkeeper monitoring provides multidimensional metrics monitoring of BookKeeper cluster as Layer: BOOKKEEPER Service in the OAP. In each cluster, the nodes are represented as Instance.

BookKeeper Cluster Supported Metrics

Monitoring Panel	Metric Name	Description	Data Source
Bookie Ledgers Count	meter_bookkeeper_bookie_ledgers_count	The number of the bookie ledgers.	Bookkeeper Cluster
Bookie Ledger Writable Dirs	meter_bookkeeper_bookie_ledger_writable_dirs	The number of writable directories in the bookie.	Bookkeeper Cluster
Bookie Ledger Dir Usage	meter_bookkeeper_bookie_ledger_dir_data_bookkeeper_ledgers_usage	The number of successfully created connections.	Bookkeeper Cluster
Bookie Entries Count	meter_bookkeeper_bookie_entries_count	The number of the bookie write entries.	Bookkeeper Cluster
Bookie Write Cache Size	meter_bookkeeper_bookie_write_cache_size	The size of the bookie write cache (MB).	Bookkeeper Cluster
Bookie Write Cache Entry Count	meter_bookkeeper_bookie_write_cache_count	The entry count in the bookie write cache.	Bookkeeper Cluster
Bookie Read Cache Size	meter_bookkeeper_bookie_read_cache_size	The size of the bookie read cache (MB).	Bookkeeper Cluster
Bookie Read Cache Entry Count	meter_bookkeeper_bookie_read_cache_count	The entry count in the bookie read cache.	Bookkeeper Cluster
Bookie Read Rate	meter_bookkeeper_bookie_read_rate	The bookie read rate (bytes/s).	Bookkeeper Cluster
Bookie Write Rate	meter_bookkeeper_bookie_write_rate	The bookie write rate (bytes/s).	Bookkeeper Cluster

BookKeeper Node Supported Metrics

Monitoring Panel	Metric Name	Description	Data Source
JVM Memory Pool Used	meter_bookkeeper_node_jvm_memory_pool_used	The usage of the broker jvm memory pool.	Bookkeeper Bookie
JVM Memory	meter_bookkeeper_node_jvm_memory_used
meter_bookkeeper_node_jvm_memory_committed
meter_bookkeeper_node_jvm_memory_init	The usage of the broker jvm memory.	Bookkeeper Bookie
JVM Threads	meter_bookkeeper_node_jvm_threads_current
meter_bookkeeper_node_jvm_threads_daemon
meter_bookkeeper_node_jvm_threads_peak
meter_bookkeeper_node_jvm_threads_deadlocked	The count of the jvm threads.	Bookkeeper Bookie
GC Time	meter_bookkeeper_node_jvm_gc_collection_seconds_sum	Time spent in a given JVM garbage collector in seconds.	Bookkeeper Bookie
GC Count	meter_bookkeeper_node_jvm_gc_collection_seconds_count	The count of a given JVM garbage.	Bookkeeper Bookie
Thread Executor Completed	meter_bookkeeper_node_thread_executor_completed	The count of the executor thread.	Bookkeeper Bookie
Thread Executor Tasks	meter_bookkeeper_node_thread_executor_tasks_completed
meter_bookkeeper_node_thread_executor_tasks_rejected
meter_bookkeeper_node_thread_executor_tasks_failed	The count of the executor tasks.	Bookkeeper Bookie
Pooled Threads	meter_bookkeeper_node_high_priority_threads
meter_bookkeeper_node_read_thread_pool_threads	The count of the pooled thread.	Bookkeeper Bookie
Pooled Threads Max Queue Size	meter_bookkeeper_node_high_priority_thread_max_queue_size
meter_bookkeeper_node_read_thread_pool_max_queue_size	The count of the pooled threads max queue size.	Bookkeeper Bookie

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in otel-rules/bookkeeper/bookkeeper-cluster.yaml, otel-rules/bookkeeper/bookkeeper-node.yaml. The Bookkeeper dashboard panel configurations are found in /config/ui-initialized-templates/bookkeeper.