Pulsar monitoring

SkyWalking leverages OpenTelemetry Collector to collect metrics data in Prometheus format from the Pulsar and transfer the metrics to OpenTelemetry receiver and into the Meter System. Kafka entity as a Service in OAP and on the `Layer: PULSAR.

Data flow

1. Pulsar exposes metrics through Prometheus endpoint.
2. OpenTelemetry Collector fetches metrics from Pulsar cluster via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
3. The SkyWalking OAP Server parses the expression with MAL to filter/calculate/aggregate and store the results.`

Setup

1. Set up Pulsar Cluster. (Pulsar cluster includes pulsar broker cluster and Bookkeeper bookie cluster.)
2. Set up OpenTelemetry Collector. The example for OpenTelemetry Collector configuration, refer to here.
3. Config SkyWalking OpenTelemetry receiver.

Pulsar Monitoring

Pulsar monitoring provides multidimensional metrics monitoring of Pulsar cluster as Layer: PULSAR Service in the OAP. In each cluster, the nodes are represented as Instance.

Pulsar Cluster Supported Metrics

Monitoring Panel	Metric Name	Description	Data Source
Total Topics	meter_pulsar_total_topics	The number of Pulsar topics in this cluster.	Pulsar Cluster
Total Subscriptions	meter_pulsar_total_subscriptions	The number of Pulsar subscriptions in this cluster.	Pulsar Cluster
Total Producers	meter_pulsar_total_producers	The number of active producers connected to this cluster.	Pulsar Cluster
Total Consumers	meter_pulsar_total_consumers	The number of active consumers connected to this cluster.	Pulsar Cluster
Message Rate In	meter_pulsar_message_rate_in	The total message rate coming into this cluster (message per second).	Pulsar Cluster
Message Rate Out	meter_pulsar_message_rate_out	The total message rate going out from this cluster (message per second).	Pulsar Cluster
Throughput In	meter_pulsar_throughput_in	The total throughput coming into this cluster (byte per second).	Pulsar Cluster
Throughput Out	meter_pulsar_throughput_out	The total throughput going out from this cluster (byte per second).	Pulsar Cluster
Storage Size	meter_pulsar_storage_size	The total storage size of all topics in this broker (in bytes).	Pulsar Cluster
Storage Logical Size	meter_pulsar_storage_logical_size	The storage size of all topics in this broker without replicas (in bytes).	Pulsar Cluster
Storage Write Rate	meter_pulsar_storage_write_rate	The total message batches (entries) written to the storage for this broker (message batch per second).	Pulsar Cluster
Storage Read Rate	meter_pulsar_storage_read_rate	The total message batches (entries) read from the storage for this broker (message batch per second).	Pulsar Cluster

Pulsar Node Supported Metrics

Monitoring Panel	Metric Name	Description	Data Source
Active Connections	meter_pulsar_broker_active_connections	The number of active connections.	Pulsar Broker
Total Connections	meter_pulsar_broker_total_connections	The total number of connections.	Pulsar Broker
Connection Create Success Count	meter_pulsar_broker_connection_create_success_count	The number of successfully created connections.	Pulsar Broker
Connection Create Fail Count	meter_pulsar_broker_connection_create_fail_count	The number of failed connections.	Pulsar Broker
Connection Closed Total Count	meter_pulsar_broker_connection_closed_total_count	The total number of closed connections.	Pulsar Broker
JVM Buffer Pool Used	meter_pulsar_broker_jvm_buffer_pool_used_bytes	The usage of jvm buffer pool.	Pulsar Broker
JVM Memory Pool Used	meter_pulsar_broker_jvm_memory_pool_used	The usage of jvm memory pool.	Pulsar Broker
JVM Memory	meter_pulsar_broker_jvm_memory_init
meter_pulsar_broker_jvm_memory_used
meter_pulsar_broker_jvm_memory_committed	The usage of jvm memory.	Pulsar Broker
JVM Threads	meter_pulsar_broker_jvm_threads_current
meter_pulsar_broker_jvm_threads_daemon
meter_pulsar_broker_jvm_threads_peak
meter_pulsar_broker_jvm_threads_deadlocked	The usage of jvm threads.	Pulsar Broker
GC Time	meter_pulsar_broker_jvm_gc_collection_seconds_sum	Time spent in a given JVM garbage collector in seconds.	Pulsar Broker
GC Count	meter_pulsar_broker_jvm_gc_collection_seconds_count	The count of a given JVM garbage collector.	Pulsar Broker

Customizations

You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in otel-rules/pulsar/pulsar-cluster.yaml, otel-rules/pulsar/pulsar-broker.yaml. The Pulsar dashboard panel configurations are found in ui-initialized-templates/pulsar.