Kubernetes (K8s) monitoring from Rover

SkyWalking uses the SkyWalking Rover system to collect access logs from Kubernetes clusters and hands them over to the OAL system for metrics and entity analysis.

Data flow

SkyWalking Rover monitoring access log data from K8s and send to the OAP.
The SkyWalking OAP Server receive access log from Rover through gRPC, analysis the generate entity, and using OAL to generating metrics.

Setup

Setup Rover in the Kubernetes and enable access log service.
Setup eBPF receiver module by the following configuration.

yaml

receiver-ebpf:
  selector: ${SW_RECEIVER_EBPF:default}
  default:

Generated Entities

SkyWalking receive the access logs from Rover, analyzes the kubernetes connection information to parse out the following corresponding entities:

Service
Service Instance
Service Endpoint
Service Relation
Service Instance Relation
Service Endpoint Relation

Generate Metrics

For each of the above-mentioned entities, metrics such as connection, transmission, and protocol can be analyzed.

Connection Metrics

Record the relevant metrics for every service establishing/closing connections with other services.

Name	Unit	Description
Connect CPM	Count	Total Connect to other Service counts per minutes.
Connect Duration	Nanoseconds	Total Connect to other Service use duration.
Connect Success CPM	Count	Success to connect to other Service counts per minutes.
Accept CPM	Count	Accept new connection from other Service counts per minutes.
Accept Duration	Nanoseconds	Total accept new connection from other Service use duration.
Close CPM	Count	Close one connection counts per minutes.
Close Duration	Nanoseconds	Total Close connections use duration.

Transfer Metrics

Record the basic information and L2-L4 layer details for each syscall made during network requests by every service to other services.

Read Data from Connection

Name	Unit	Description
Read CPM	Count	Read from connection counts per minutes.
Read Duration	Nanoseconds	Total read data use duration.
Read Package CPM	Count	Total read TCP Package count per minutes.
Read Package Size	Bytes	Total read TCP package size per minutes.
Read Layer 4 Duration	Nanoseconds	Total read data on the Layer 4 use duration.
Read Layer 3 Duration	Nanoseconds	Total read data on the Layer 3 use duration.
Read Layer 3 Recv Duration	Nanoseconds	Total read data on the Layer 3 receive use duration.
Read Layer 3 Local Duration	Nanoseconds	Total read data on the Layer 3 local use duration.
Read Package To Queue Duration	Nanoseconds	Total duration between TCP package received and send to Queue.
Read Package From Queue Duration	Nanoseconds	Total duration between send to Queue and receive from Queue.
Read Net Filter CPM	Count	Total Net Filtered count when read data.
Read Net Filter Duration	Nanoseconds	Total Net Filtered use duration.

Write Data to Connection

Name	Unit	Description
Write CPM	Count	Write to connection counts per minutes.
Write Duration	Nanoseconds	Total write data to connection use duration.
Write Package CPM	Count	Total write TCP Package count per minutes.
Write Package Size	Bytes	Total write TCP Package size per minutes.
Write L4 Duration	Nanoseconds	Total write data to connection Layer 4 use duration.
Write L3 Duration	Nanoseconds	Total write data to connection Layer 3 use duration.
Write L3 Local Duration	Nanoseconds	Total write data to the connection Layer 3 Local use duration.
Write L3 Output Duration	Nanoseconds	Total write data to the connection Layer 3 Output use duration.
Write L2 Duration	Nanoseconds	Total write data to connection Layer 2 use duration.
Write L2 Ready Send Duration	Nanoseconds	Total write data to the connection Layer 2 ready send data queue use duration.
Write L2 Send NetDevice Duration	Nanoseconds	Total write data to the connection Layer 2 send data to net device use duration.

Protocol

Based on each transfer data analysis, extract the information of the 7-layer network protocol.

HTTP/1.x or HTTP/2.x

Name	Init	Description
Call CPM	Count	HTTP Request calls per minutes.
Duration	Milliseconds	Total HTTP Response use duration.
Success CPM	Count	Total HTTP Response success(status < 500) count.
Request Header Size	Bytes	Total Request Header size.
Request Body Size	Bytes	Total Request Body size.
Response Header Size	Bytes	Total Response Header size.
Response Body Size	Bytes	Total Response Body size.

Customizations

You can customize your own metrics/dashboard panel. The metrics definition and expression rules are found in /config/oal/ebpf.oal, please refer the Scope Declaration Documentation. The K8s dashboard panel configurations are found in /config/ui-initialized-templates/k8s_service.