docs/en/setup/backend/backend-elasticsearch-monitoring.md
SkyWalking leverages elasticsearch-exporter for collecting metrics data from Elasticsearch. It leverages OpenTelemetry Collector to transfer the metrics to OpenTelemetry receiver and into the Meter System.
Elasticsearch monitoring provides multidimensional metrics monitoring of Elasticsearch clusters as Layer: ELASTICSEARCH Service in the OAP. In each cluster, the nodes are represented as Instance and indices are Endpoints.
| Monitoring Panel | Metric Name | Description | Data Source |
|---|---|---|---|
| Cluster Health | meter_elasticsearch_cluster_health_status | Whether all primary and replica shards are allocated | elasticsearch-exporter |
| Tripped Of Breakers | meter_elasticsearch_cluster_breakers_tripped | Tripped for breaker | elasticsearch-exporter |
| Nodes | meter_elasticsearch_cluster_nodes | Number of nodes in the cluster. | elasticsearch-exporter |
| Data Nodes | meter_elasticsearch_cluster_data_nodes | Number of data nodes in the cluster | elasticsearch-exporter |
| Pending Tasks | meter_elasticsearch_cluster_pending_tasks_total | Cluster level changes which have not yet been executed | elasticsearch-exporter |
| CPU Usage Avg. (%) | meter_elasticsearch_cluster_cpu_usage_avg | Cluster level percent CPU used by process | elasticsearch-exporter |
| JVM Memory Used Avg. (%) | meter_elasticsearch_cluster_jvm_memory_used_avg | Cluster level percent JVM memory used | elasticsearch-exporter |
| Open Files | meter_elasticsearch_cluster_open_file_count | Open file descriptors | elasticsearch-exporter |
| Active Primary Shards | meter_elasticsearch_cluster_primary_shards_total | The number of primary shards in your cluster. This is an aggregate total across all indices | elasticsearch-exporter |
| Active Shards | meter_elasticsearch_cluster_shards_total | Aggregate total of all shards across all indices, which includes replica shards | elasticsearch-exporter |
| Initializing Shards | meter_elasticsearch_cluster_initializing_shards_total | Count of shards that are being freshly created | elasticsearch-exporter |
| Delayed Unassigned Shards | meter_elasticsearch_cluster_delayed_unassigned_shards_total | Shards delayed to reduce reallocation overhead | elasticsearch-exporter |
| Relocating Shards | meter_elasticsearch_cluster_relocating_shards_total | The number of shards that are currently moving from one node to another node | elasticsearch-exporter |
| Unassigned Shards | meter_elasticsearch_cluster_unassigned_shards_total | The number of shards that exist in the cluster state, but cannot be found in the cluster itself | elasticsearch-exporter |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Node Rules | meter_elasticsearch_node_rules | Node roles | elasticsearch-exporter | |
| JVM Memory Used | MB | meter_elasticsearch_node_jvm_memory_used | Node level JVM memory used size | elasticsearch-exporter |
| CPU Percent | % | meter_elasticsearch_node_process_cpu_percent | Node level percent CPU used by process | elasticsearch-exporter |
| Documents | meter_elasticsearch_node_indices_docs | Count of index documents on this node | elasticsearch-exporter | |
| Segments | meter_elasticsearch_node_segment_count | Count of index segments on this node | elasticsearch-exporter | |
| Disk Free Space | GB | meter_elasticsearch_node_all_disk_free_space | Available space on all block device | elasticsearch-exporter |
| Open Files | meter_elasticsearch_node_open_file_count | Open file descriptors | elasticsearch-exporter | |
| Process CPU Usage Percent | % | meter_elasticsearch_node_process_cpu_percent | Percent CPU used by process | elasticsearch-exporter |
| OS CPU usage percent | % | meter_elasticsearch_node_os_cpu_percent | Percent CPU used by the OS | elasticsearch-exporter |
| Load Average | meter_elasticsearch_node_os_load1 | |||
| meter_elasticsearch_node_os_load5 | ||||
| meter_elasticsearch_node_os_load15 | Shortterm, Midterm, Longterm load average | elasticsearch-exporter | ||
| JVM Memory Usage | MB | meter_elasticsearch_node_jvm_memory_nonheap_used | ||
| meter_elasticsearch_node_jvm_memory_heap_used | ||||
| meter_elasticsearch_node_jvm_memory_heap_max | JVM memory currently usage by area | elasticsearch-exporter | ||
| JVM Pool Peak Used | MB | meter_elasticsearch_node_jvm_memory_pool_peak_used | JVM memory currently used by pool | elasticsearch-exporter |
| GC Count | meter_elasticsearch_node_jvm_gc_count | Count of JVM GC runs | elasticsearch-exporter | |
| GC Time | ms/min | meter_elasticsearch_node_jvm_gc_time | GC run time | elasticsearch-exporter |
| All Operations ReqRate | meter_elasticsearch_node_indices_*_req_rate | All Operations ReqRate on node | elasticsearch-exporter | |
| Indexing Rate | reqps | meter_elasticsearch_node_indices_indexing_index_total_req_rate | ||
| meter_elasticsearch_node_indices_indexing_index_total_proc_rate | Indexing rate on node | elasticsearch-exporter | ||
| Searching Rate | reqps | meter_elasticsearch_node_indices_search_fetch_total_req_rate | ||
| meter_elasticsearch_node_indices_search_query_time_seconds_proc_rate | Searching rate on node | elasticsearch-exporter | ||
| Total Translog Operations | meter_elasticsearch_node_indices_translog_operations | Total translog operations | elasticsearch-exporter | |
| Total Translog Size | MB | meter_elasticsearch_node_indices_translog_size | Total translog size | elasticsearch-exporter |
| Tripped For Breakers | meter_elasticsearch_node_breakers_tripped | Tripped for breaker | elasticsearch-exporter | |
| Estimated Size Of Breaker | MB | meter_elasticsearch_node_breakers_estimated_size | Estimated size of breaker | elasticsearch-exporter |
| Documents Count | KB/s | meter_elasticsearch_node_indices_docs | Count of documents on this node | elasticsearch-exporter |
| Merged Documents Count | count/s | meter_elasticsearch_node_indices_merges_docs_total | Cumulative docs merged | elasticsearch-exporter |
| Deleted Documents Count | meter_elasticsearch_node_indices_docs_deleted_total | Count of deleted documents on this node | elasticsearch-exporter | |
| Documents Index Rate | calls/s | meter_elasticsearch_node_indices_indexing_index_total_req_rate | Total index calls per second | elasticsearch-exporter |
| Merged Documents Rate | MB / s | meter_elasticsearch_node_indices_merges_total_size_bytes_total | Total merge size per second | elasticsearch-exporter |
| Documents Deleted Rate | docs/s | meter_elasticsearch_node_indices_docs_deleted | Count of deleted documents per second on this node | elasticsearch-exporter |
| Count Of Index Segments | meter_elasticsearch_node_segment_count | Count of index segments on this node | elasticsearch-exporter | |
| Current Memory Size Of Segments | MB | meter_elasticsearch_node_segment_memory | Current memory size of segments | elasticsearch-exporter |
| Network | bytes/sec | meter_elasticsearch_node_network_send_bytes | ||
| meter_elasticsearch_node_network_receive_bytes | Total number of bytes sent and receive | elasticsearch-exporter | ||
| Disk Usage Percent | % | meter_elasticsearch_node_disk_usage_percent | Used space on block device | elasticsearch-exporter |
| Disk Usage | GB | meter_elasticsearch_node_disk_usage | Used space size of block device | elasticsearch-exporter |
| Disk Read | KBs | meter_elasticsearch_node_disk_io_read_bytes | Total kilobytes read from disk | elasticsearch-exporter |
| Disk Write | KBs | meter_elasticsearch_node_disk_io_write_bytes | Total kilobytes write from disk | elasticsearch-exporter |
| Monitoring Panel | Unit | Metric Name | Description | Data Source |
|---|---|---|---|---|
| Documents Primary | meter_elasticsearch_index_indices_docs_primary | Count of documents with only primary shards on all nodes | elasticsearch-exporter | |
| Deleted Documents Primary | meter_elasticsearch_index_indices_deleted_docs_primary | Count of deleted documents with only primary shards | elasticsearch-exporter | |
| Data Primary | GB | meter_elasticsearch_index_indices_store_size_bytes_primary | Current total size of stored index data with only primary shards on all nodes | elasticsearch-exporter |
| Data | GB | meter_elasticsearch_index_indices_store_size_bytes_total | Current total size of stored index data with all shards on all nodes | elasticsearch-exporter |
| Segments Primary | meter_elasticsearch_index_indices_segment_count_primary | Current number of segments with only primary shards on all nodes | elasticsearch-exporter | |
| Segments Memory Primary | MB | meter_elasticsearch_index_indices_segment_memory_bytes_primary | Current size of segments with only primary shards on all nodes | elasticsearch-exporter |
| Segments | meter_elasticsearch_index_indices_segment_count_total | Current number of segments with all shards on all nodes | elasticsearch-exporter | |
| Segments Memory | MB | meter_elasticsearch_index_indices_segment_memory_bytes_total | Current size of segments with all shards on all nodes | elasticsearch-exporter |
| Indexing Rate | meter_elasticsearch_index_stats_indexing_index_total_req_rate | |||
| meter_elasticsearch_index_stats_indexing_index_total_proc_rate | Indexing rate on index | elasticsearch-exporter | ||
| Searching Rate | meter_elasticsearch_index_stats_search_query_total_req_rate | |||
| meter_elasticsearch_index_stats_search_query_total_proc_rate | Searching rate on index | elasticsearch-exporter | ||
| All Operations ReqRate | meter_elasticsearch_index_stats_*_req_rate | All Operations ReqRate on index | elasticsearch-exporter | |
| All Operations Runtime | meter_elasticsearch_index_stats_*_time_seconds_total | All Operations Runtime/s on index | elasticsearch-exporter | |
| Avg. Search Time Execute / Request | s | meter_elasticsearch_index_search_fetch_avg_time | ||
| meter_elasticsearch_index_search_query_avg_time | ||||
| meter_elasticsearch_index_search_scroll_avg_time | ||||
| meter_elasticsearch_index_search_suggest_avg_time | Search Operation Avg. time on index | elasticsearch-exporter | ||
| Search Operations Rate | req/s | meter_elasticsearch_index_stats_search_query_total_req_rate | ||
| meter_elasticsearch_index_stats_search_fetch_total_req_rate | ||||
| meter_elasticsearch_index_stats_search_scroll_total_req_rate | ||||
| meter_elasticsearch_index_stats_search_suggest_total_req_rate | Search Operations ReqRate on index | elasticsearch-exporter | ||
| Shards Documents | meter_elasticsearch_index_indices_shards_docs | Count of documents per shards on index | elasticsearch-exporter | |
| Documents (Primary Shards) | meter_elasticsearch_index_indices_docs_primary | Count of documents with only primary shards on index | elasticsearch-exporter | |
| Documents Created Per Min (Primary Shards) | meter_elasticsearch_index_indices_docs_primary_rate | Documents rate with only primary shards on index | elasticsearch-exporter | |
| Total Size Of Index (Primary Shards) | MB | meter_elasticsearch_index_indices_store_size_bytes_primary | Current total size of stored index data in bytes with only primary shards on all nodes | elasticsearch-exporter |
| Documents (All Shards) | meter_elasticsearch_index_indices_docs_total | Count of documents with all shards on index | elasticsearch-exporter | |
| Documents Created Per Min (All Shards) | meter_elasticsearch_index_indices_docs_total_rate | Documents rate with only all shards on index | elasticsearch-exporter | |
| Total Size Of Index (All Shards) | MB | meter_elasticsearch_index_indices_store_size_bytes_total | Current total size of stored index data in bytes with all shards on all nodes | elasticsearch-exporter |
You can customize your own metrics/expression/dashboard panel.
The metrics definition and expression rules are found
in /config/otel-rules/elasticsearch/elasticsearch-cluster.yaml, /config/otel-rules/elasticsearch/elasticsearch-node.yaml, /config/otel-rules/elasticsearch/elasticsearch-index.yaml.
The Elasticsearch dashboard panel configurations are found in /config/ui-initialized-templates/elasticsearch.