Lookup Processor Plugin

This plugin allows to use one or more files containing lookup-tables for annotating incoming metrics. The lookup is static as the files are only used on startup. The main use-case for this is to annotate metrics with additional tags e.g. dependent on their source. Multiple tags can be added depending on the lookup-table files.

The lookup key can be generated using a Golang template with the ability to access the metric name via {{.Name}}, the tag values via {{.Tag "mytag"}}, with mytag being the tag-name and field-values via {{.Field "myfield"}}, with myfield being the field-name. Non-existing tags and field will result in an empty string or nil respectively. In case the key cannot be found, the metric is passed-through unchanged. By default all matching tags are added and existing tag-values are overwritten.

[!NOTE] The plugin only supports the addition of tags and thus all mapped tag-values need to be strings!

⭐ Telegraf v1.15.0 🏷️ annotation 💻 all

Global configuration options <!-- @/docs/includes/plugin_config.md -->

Plugins support additional global and plugin configuration settings for tasks such as modifying metrics, tags, and fields, creating aliases, and configuring plugin ordering. See CONFIGURATION.md for more details.

Configuration

# Lookup a key derived from metrics in a static file
[[processors.lookup]]
  ## List of files containing the lookup-table
  files = ["path/to/lut.json", "path/to/another_lut.json"]

  ## Format of the lookup file(s)
  ## Available formats are:
  ##    json               -- JSON file with 'key: {tag-key: tag-value, ...}' mapping
  ##    csv_key_name_value -- CSV file with 'key,tag-key,tag-value,...,tag-key,tag-value' mapping
  ##    csv_key_values     -- CSV file with a header containing tag-names and
  ##                          rows with 'key,tag-value,...,tag-value' mappings
  # format = "json"

  ## Template for generating the lookup-key from the metric.
  ## This is a Golang template (see https://pkg.go.dev/text/template) to
  ## access the metric name (`{{.Name}}`), a tag value (`{{.Tag "name"}}`) or
  ## a field value (`{{.Field "name"}}`).
  key = '{{.Tag "host"}}'

File formats

The following descriptions assume keys to be unique identifiers used for matching the configured key. The tag-name/tag-value pairs are the tags added to a metric if the key matches.

json format

In the json format, the input files must have the following format

{
  "keyA": {
    "tag-name1": "tag-value1",
    ...
    "tag-nameN": "tag-valueN",
  },
  ...
  "keyZ": {
    "tag-name1": "tag-value1",
    ...
    "tag-nameM": "tag-valueM",
  }
}

Please note that only strings are supported for all elements.

csv_key_name_value format

The csv_key_name_value format specifies comma-separated-value files with the following format

# Optional comments
keyA,tag-name1,tag-value1,...,tag-nameN,tag-valueN
keyB,tag-name1,tag-value1
...
keyZ,tag-name1,tag-value1,...,tag-nameM,tag-valueM

The formatting uses commas (,) as separators and allows for comments defined as lines starting with a hash (#). All lines can have different numbers but must at least contain three columns and follow the name/value pair format. There cannot be a name without value.

csv_key_values format

This setting specifies comma-separated-value files with the following format

# Optional comments
ignored,tag-name1,...,tag-nameN
keyA,tag-value1,...,,,,
keyB,tag-value1,,,,...,
...
keyZ,tag-value1,...,tag-valueM,...,

The formatting uses commas (,) as separators and allows for comments defined as lines starting with a hash (#). All lines must contain the same number of columns. The first non-comment line must contain a header specifying the tag-names. As the first column contains the key to match the first header value is ignored. There have to be at least two columns.

Please note that empty tag-values will be ignored and the tag will not be added.

Example

With a lookup table of

{
  "xyzzy-green": {
    "location": "eu-central",
    "rack": "C12-01"
  },
  "xyzzy-red": {
    "location": "us-west",
    "rack": "C01-42"
  },
}

in format = "json" and a key of key = '{{.Name}}-{{.Tag "host"}}' you get

- xyzzy,host=green value=3.14 1502489900000000000
- xyzzy,host=red  value=2.71 1502499100000000000
+ xyzzy,host=green,location=eu-central,rack=C12-01 value=3.14 1502489900000000000
+ xyzzy,host=red,location=us-west,rack=C01-42 value=2.71 1502499100000000000
xyzzy,host=blue  value=6.62 1502499700000000000

The same results can be achieved with format = "csv_key_name_value" and

xyzzy-green,location,eu-central,rack,C12-01
xyzzy-red,location,us-west,rack,C01-42

or format = "csv_key_values" and

-,location,rack
xyzzy-green,eu-central,C12-01
xyzzy-red,us-west,C01-42