0.35 Upgrade Guide

Vector's 0.35.0 release includes breaking changes:

1. The Throttle transform's events_discarded_total internal metric is now opt-in
2. The file internal metric tag is now opt-in for file-based components
3. Datadog sinks now default the API key and Site to the value of environment variables
4. Removal of requests_completed_total, request_duration_seconds, and requests_received_total Internal Metrics

and deprecations:

1. Deprecation of undefined environment variable warnings

and potentially impactful changes:

1. HTTP server-based sources now include a keepalive.max_connection_age_secs config option that defaults to 5 minutes
2. Update component_sent_bytes_total to correctly report uncompressed bytes for all sinks
3. Update component_received_bytes_total to correctly report decompressed bytes for all sources
4. Update default values for the request.retry_max_duration_secs and request.rate_limit_num sink configuration options

We cover them below to help you upgrade quickly:

Upgrade guide

Breaking Changes

The Throttle transform's events_discarded_total internal metric is now opt-in {#events-discarded-total-opt-in}

The Throttle transform's events_discarded_total internal metric, which includes the key tag, is now only emitted on an opt-in basis. Users can opt-in to emit this metric by setting internal_metrics.emit_events_discarded_per_key to true in the corresponding Throttle transform component config. This change is motivated by the fact that the key metric tag has potentially unbounded cardinality.

To view events discarded without the key tag, use the component_discarded_events_total internal metric.

The file internal metric tag is now opt-in for file-based components {#file-tag-opt-in}

File-based components (file source, Kubernetes logs source, file sink) now include a internal_metrics.include_file_tag config option that determines whether the file tag is included on the component's corresponding internal metrics. This config option defaults to false, as this tag is likely to be of high cardinality.

Removal of requests_completed_total, request_duration_seconds, and requests_received_total Internal Metrics {#remove-obsolete-http-metrics}

The requests_completed_total, request_duration_seconds, and requests_received_total internal metrics were removed in the 0.35.0 release.

Removed metric	Replacement metric	Affected components
requests_completed_total	http_client_responses_total	apache metrics, aws ecs metrics, http client, prometheus remote write, prometheus scrape
request_duration_seconds	http_client_response_rtt_seconds	apache metrics, aws ecs metrics, http client, prometheus remote write, prometheus scrape
requests_received_total	http_server_requests_received_total	aws kinesis firehose, heroku logplex, prometheus exporter, splunk hec

Deprecations

Deprecation of undefined environment variable warnings {#strict-env-vars}

Vector has the ability to interpolate environment variables into its configuration as a means of templating configuration. Currently, if an environment variable is undefined, unless the ? interpolation syntax is used, Vector outputs a warning and replaces the environment variable with an empty string. In a future release , this will change from a warning to a hard error. This behavior can be opted into earlier by using the --strict-env-vars flag (or VECTOR_STRICT_ENV_VARS environment variable) when running Vector.

The reasoning for this change is that users often miss the undefined variable warning and are confused by Vector not behaving as the expected. In particular, this happens when using $1 in regex capture groups in VRL without realizing they need to be escaped as $$1 to avoid interpolation.

Potentially impactful changes

HTTP server-based sources now include a keepalive.max_connection_age_secs config option that defaults to 5 minutes {#http-keepalive-max-connection-age}

HTTP server-based sources include a new keepalive.max_connection_age_secs configuration option, which defaults to 5 minutes (300 seconds). When enabled, this closes incoming TCP connections that reach the maximum age by sending a Connection: close header in the response. While this parameter is crucial for managing the lifespan of persistent, incoming connections to Vector and for effective load balancing, it can be disabled by setting keepalive.max_connection_age_secs to a very large number like 100000000 (ideally you could set it to null, but Vector currently requires all values to be valid TOML values, of which null is not).

Update component_sent_bytes_total to correctly report uncompressed bytes for all sinks {#component-sent-bytes-total}

The AppSignal, Datadog Metrics, GreptimeDB, GCP Cloud Monitoring, Honeycomb, and HTTP sinks now correctly report uncompressed bytes, rather than compressed bytes, for the component_sent_bytes_total internal metric.

Update component_received_bytes_total to correctly report decompressed bytes for all sources {#component-received-bytes-total}

The Heroku Logs, HTTP Server, Prometheus Remote Write, and Splunk HEC sources now correctly report decompressed bytes, rather than compressed bytes, for the component_received_bytes_total internal metric.

Update default values for the request.retry_max_duration_secs and request.rate_limit_num sink configuration options {#request-config-options}

The request.retry_max_duration_secs config option previously defaulted to 3600 seconds. It now defaults to 30 seconds.

Also, a bug was fixed that prevented component-level default values from being applied. In particular, this updates the default value for request.rate_limit_num for the following sinks:

Sink	Default request.rate_limit_num Value	Previous Limit
AWS Cloudwatch Metrics Sink	150	No limit
Azure Blob Storage Sink	250	No limit
GCP Chronicle Unstructured Sink	1000	No limit
GCP Cloud Storage Sink	1000	No limit
GCP Cloud Monitoring Sink	1000	No limit

Note that all changes described above are reflected in the component reference documentation.

Datadog sinks now default the API key and Site to the value of environment variables {#datadog-env-vars}

The Datadog Logs, Metrics, and Traces sinks now default the values of the default_api_key and site configuration options to the values of environment variables DD_API_KEY and DD_SITE, respectively.