libbeat/autodiscover/README.md
https://www.elastic.co/guide/en/beats/metricbeat/master/configuration-autodiscover.html https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html https://www.elastic.co/guide/en/fleet/7.x/dynamic-input-configuration.html#providers
When you run applications on containers, they become moving targets to the monitoring system. Autodiscover allows you to track them and adapt settings as changes happen. By defining configuration templates, the autodiscover subsystem can monitor services as they start running.
You define autodiscover settings in the autodiscover section of the metricbeat.yml/filebeat.yml config file. To enable autodiscover, you specify a list of providers. For Elastic Agent we will discuss later.
Providers Autodiscover providers work by watching for events on the system and translating those events into internal autodiscover events with a common format. When you configure the provider, you can optionally use fields from the autodiscover event to set conditions that, when met, launch specific configurations.
On start Metricbeat/Filebeat/Agent will scan existing containers and launch the proper configs for them. Then it will watch for new start/stop events. This ensures you don’t need to worry about state, but only define your desired configs.
The Kubernetes autodiscover provider watches for Kubernetes nodes, pods, services and namespaces when they start, update, and stop.
We will describe the internals of three ways of Kubernetes autodiscover process.
As the name suggests, user needs to set a template to indicate to autodiscover provider what to do.
There is one configuration variable that differentiates in a way how the autodiscover process is performed.
This variable is unique
When setting unique: true the Leader Election mechanism is activated. That way only the Beat instance that will gain the leader lease/lock will enable the provided template.
The best appliance of this feature is when collecting cluster wide metrics from kube-state-metrics or apiserver.
In that case having all instances of metricbeat collecting the same metrics is not desirable.
Example:
metricbeat.autodiscover:
providers:
- type: kubernetes
scope: cluster
node: ${NODE_NAME}
unique: true
identifier: leader-election-metricbeat
templates:
- config:
- module: kubernetes
hosts: ["kube-state-metrics:8080"]
period: 10s
add_metadata: true
metricsets:
- state_node
We will deep dive in the internals of libbeat kubernetes autodiscover provider.
We will use metricbeat as an example.
Step-by-step walkthrough
init function adds the provider in the autodiscover providers registry at startup. For Kubernetes provider an AutodiscoverBuilder func is passed as an argument.NewAutodiscover function which checks in the config for enabled providers and builds them one by one, calling the AutodiscoverBuilder func.AutodiscoverBuilder creates and returns a Kubernetes Provider struct which is then added to an Autodiscover manager struct.OnStartedLeading is executed when the specific metricbeat instance gains the leader election lock. StartLeading creates a bus event with "start": true, and publishes it. The template configurations is also added in this event.updated.When setting unique: false or not setting it at all the Leader Election mechanism is disabled. That way all Beat instances will enable the provided template. Or at least the ones that match a condition.
A good appliance of this is when user wants to enable a specific module(for example redis) each time a pod with a specific label appears in each kubernetes node.
Example:
metricbeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
contains:
kubernetes.labels.app: "redis"
config:
- module: redis
metricsets: ["info", "keyspace"]
hosts: "${data.host}:6379"
password: "${REDIS_PASSWORD}"
We will use metricbeat and redis module as example. Step-by-step walkthrough
Steps 1-3 are exactly the same as with Leader Election.
4. In the Kubernetes Provider struct created by the Kubernetes AutodiscoverBuilder in case there is a condition a NewConfigMapper is created. It contains the condition map with all the conditions of a given config.
4. When unique is set to false, the resource is checked from the configuration template(default is pod) and a new PodEventer/NodeEventer/ServiceEventer is set as the eventManager of Kubernetes Provider.
5. The kubernetes node that the metricbeat instance is running on is discovered.
6. A dedicated watcher to watch pods, nodes and namespaces get started. Also a metadata generator is started to enrich the events with kubernetes metadata.
7. Each time a new event is about to be published by the watchers(we will have a dedicated section of how events are published), it is checked whether the condition set by the user matches the event.
8. If it matches, then a config is created with the configuration set by user and it is added in the event. After that the event gets published.
9. The worker listening for events receives the start event and checks if there is any update in the configs of the event. (same process with leader election steps 6-9)
10. If there is an update it starts or stops the runners needed. In our example it starts a new runner that starts the redis module.
The watcher struct has a kubernetes informer field which depending on the resource type watches and lists the changes in the kubernetes cluster for that specific resource. When a new pod for example is discovered it is added in the work queue for processing. The watcher processes that queue and depending if there was an addition,deletion or update of a resource it calls the related method.
start flag. Emit adds the pod to a new event together will all metadata, sets start: true and publishes it. Step 7 follows.stop flag. Emit adds the pod to a new event with stop: true and publishes it. Step 7 follows.stop flag to generate a stop event and then with start flag to generate a start event.https://www.elastic.co/guide/en/beats/metricbeat/master/configuration-autodiscover-hints.html
Metricbeat, Filebeat and Heartbeat support autodiscover based on hints from the provider. The hints system looks for hints in Kubernetes Pod annotations which have the prefix co.elastic.metrics. As soon as the container starts, Metricbeat/Filebeat will check if it contains any hints and launch the proper config for it. Hints tell Metricbeat/Filebeat how to get metrics for the given container.
Example metricbeat configuration:
metricbeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
Example redis pod with right annotations:
apiVersion: v1
kind: Pod
metadata:
name: redis
annotations:
co.elastic.metrics/module: redis
co.elastic.metrics/metricsets: info
co.elastic.metrics/hosts: '${data.host}:6379'
spec:
containers:
- name: redis
image: redis:5.0.4
command:
- redis-server
- "/redis-master/redis.conf"
env:
- name: MASTER
value: "true"
ports:
- containerPort: 6379
resources:
limits:
cpu: "0.1"
volumeMounts:
- mountPath: /redis-master-data
name: data
- mountPath: /redis-master
name: config
volumes:
- name: data
emptyDir: {}
- name: config
configMap:
name: example-redis-config
items:
- key: redis-config
path: redis.conf
Everything works the same as Autodiscover without LeaderElection until step 8.
Autodiscover without LeaderElection step 8.
They contain the same information as if they were set explicitly in the Metricbeat configuration but actually derive from the pod annotations.Autodiscover without LeaderElection step 9 and onward.Follow the link here.