Template syntax

Vector supports a template syntax for some configuration options. This allows for dynamic values derived from event data. Any option that supports this syntax will be clearly documented as such in the description.

Example

Let's partition data on AWS S3 by "application_id" and "date". We can accomplish this with the key_prefix option in the aws_s3 sink:

{{< tabs default="YAML" >}} {{< tab title="YAML" >}}

sinks:
  backup:
    type: "aws_s3"
    bucket: "all_application_logs"
    key_prefix: "application_id={{ application_id }}/date=%F/"

{{< /tab >}} {{< tab title="TOML" >}}

[sinks.backup]
  type = "aws_s3"
  bucket = "all_application_logs"
  key_prefix = "application_id={{ application_id }}/date=%F/"

{{< /tab >}} {{< /tabs >}}

Notice that Vector allows direct field references as well as "strftime" specifiers. If we were to run the following log event through Vector:

{
  "timestamp": "2020-02-14T01:22:23.223Z",
  "application_id": 1,
  "message": "Hello world"
}

The value of the above key_prefix option would equal:

application_id=1/date=2020-02-14

Because the aws_s3 sink batches data, each event would be grouped by its produced value. This effectively enables dynamic partitioning, something fundamental to storing log data in filesystems.

Syntax

Event fields

Individual log event fields can be accessed using {{ ... }} to wrap a VRL path expression:

{{< tabs default="YAML" >}} {{< tab title="YAML" >}}

option: "{{ .parent.child }}"

{{< /tab >}} {{< tab title="TOML" >}}

option = "{{ .parent.child }}"

{{< /tab >}} {{< /tabs >}}

Strftime specifiers

In addition to directly accessing fields, Vector offers a shortcut for injecting strftime specifiers:

{{< tabs default="YAML" >}} {{< tab title="YAML" >}}

option: "year=%Y/month=%m/day=%d/"

{{< /tab >}} {{< tab title="TOML" >}}

option = "year=%Y/month=%m/day=%d/"

{{< /tab >}} {{< /tabs >}}

{{< info >}} The value is derived from the timestamp field and the name of this field can be changed via the global timestamp_key option. {{< /info >}}

Escaping

You can escape this syntax by prefixing the character with a \. For example, you can escape the event field syntax like this:

{{< tabs default="YAML" >}} {{< tab title="YAML" >}}

option: '\{{ field_name }}'

{{< /tab >}} {{< tab title="TOML" >}}

option = "\{{ field_name }}"

{{< /tab >}} {{< /tabs >}}

And strftime specified like so:

{{< tabs default="YAML" >}} {{< tab title="YAML" >}}

option: "year=\\%Y/month=\\%m/day=\\%d/"

{{< /tab >}} {{< tab title="TOML" >}}

option = "year=\%Y/month=\%m/day=\%d/"

{{< /tab >}} {{< /tabs >}}

Each of the values above would be treated literally.

How it works

Accessing fields

You can find additional examples for accessing fields in the path expression reference documentation.

Fallback values

Vector doesn't currently support fallback values, issue 1692 is open to add this functionality. In the interim, you can use the remap transform to set a default value:

{{< tabs default="YAML" >}} {{< tab title="YAML" >}}

transforms:
  set_defaults:
    type: "remap"
    inputs:
      - "my-source-id"
    source: |
      if !exists(.my_field) {
        .my_field = "default"
      }

{{< /tab >}} {{< tab title="TOML" >}}

[transforms.set_defaults]
  type = "remap"
  inputs = ["my-source-id"]
  source = '''
    if !exists(.my_field) {
      .my_field = "default"
    }
  '''

{{< /tab >}} {{< /tabs >}}

Missing fields

If a field is missing, an error is logged and Vector drops the event. The component_errors_total internal metric is incremented with an error_type tag of template_failed.