0.12 Upgrade Guide

0.12 includes minimal breaking changes but significant deprecations. This guide will upgrade you quickly and painlessly. If you have questions, hop in our chat and we'll help you upgrade.

1. Breaking: The encoding.codec option is now required for all relevant sinks
2. Breaking: Vector check_fields conditions now require the type option
3. Breaking: The generator source requires a format option
4. Deprecation: Many transforms have been deprecated in favor of the new remap transform
5. Deprecation: The file source start_at_beginning has been deprecated

Upgrade Guide

Breaking: The encoding.codec option is now required for all relevant sinks<a name="first"></a>

Pull request #5281 removed the default values for the sink-level encoding.codec option. Therefore, you are now required to provide a value for this option, ensuring that you are not surprised by opinionated encoding defaults. Affected sinks include:

• aws_s3 (previously defaulted to text)
• file (previously defaulted to text)
• humio (previously defaulted to json)
• kafka (previously defaulted to text)
• nats (previously defaulted to text)
• new_relic_logs (previously defaulted to json)
• pulsar (previously defaulted to text)
• splunk_hec (previously defaulted to text)

Upgrading is easy, just add the encoding.codec to your sinks with your preferred format (json or text):

 [sinks.backup]
 type = "aws_s3"
 inputs = ["..."]
 bucket = "my-bucket"
 compression = "gzip"
 region = "us-east-1"
+encoding.codec = "json"

For clarity, the text option strips away all structured data and passes only the value of the message field. It is intended for use cases where Vector acts as a proxy and should not alter data. For most use cases we recommend json since it includes all structured data.

Breaking: Vector check_fields conditions now require the type option<a name="second"></a>

With the announcement of the Vector Remap Language (VRL), pull request #5978 deprecated the check_fields conditions in favor of using VRL boolean expressions. The old check_fields conditions were limiting and suffered from many of the pitfalls outlined in the VRL announcement. Configuration languages, like TOML, are bad at expressing boolean conditions and severely limited how users could route, filter, and reduce data.

While check_fields is deprecated and still supported, you will need to explicitly opt-into the feature by adding the type option:

 [transforms.route]
 type = "route"
+lanes.errors.type = "check_field"
 lanes.errors."level.eq" = "error"

Alteratively, we recommend migrating to the new VRL syntax:

 [transforms.route]
 type = "route"
-lanes.errors."level.eq" = "error"
+lanes.errors = '.level = "error"'

Refer to the VRL reference for the many ways you can specify conditions.

Breaking: The generator source requires a format option<a name="third"></a>

The [generator source], commonly used for testing, has been updated with a new format option that emits logs in the specified format. You will not be required to provide this option. Upgrading is easy:

 [sources.generator]
 type = "generator"
+format = "apache_common"  # or "apache_error" or "syslog"

Deprecation: Many transforms have been deprecated in favor of the new remap transform<a name="fourth"></a>

The following transforms have been deprecated in favor of the new [remap transform][remap_transform]:

• add_fields
• add_tags
• ansi_stripper
• aws_cloudwatch_logs_subscription_parser
• coercer
• concat
• grok_parser
• json_parser
• key_value_parser
• logfmt_parser
• merge
• regex_parser
• remove_fields
• remove_tags
• rename_fields
• split
• tokenizer

Deprecation notices have been placed on each of these transforms with example VRL programs that demonstrate how to migrate to the new remap transform. For example, migrating from the json_parser transform is as simple as:

transforms:
  remap:
    type: "remap"
    source: |
      . = merge(., parse_json!(.message))

You do not need to upgrade immediately. These transforms will not be removed until Vector hits 1.0, a milestone that we hope to achieve in late 2022. But, if possible, we recommend using this opportunity to upgrade and significantly simplify your Vector configuration.

As always, if you need assistance hop in our chat. We're eager to help and receive feedback on the language.

Deprecation: The file source start_at_beginning has been deprecated<a name="fifth"></a>

As noted in the file source checkpointing highlight, we've removed the start_at_beginning option and replaced it with new ignore_checkpoints and read_from options. Migrating is easy:

 [sources.file]
 type = "file"
-start_at_beginning = true
+ignore_checkpoints = false # default
+read_from = "beginning" # default

Adjust as necessary. The above values are the defaults and are not required to be specified.