Vector v0.25.0 release notes

The Vector team is pleased to announce version 0.25.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In addition to the new features, enhancements, and fixes listed below, this release adds:

  • A new http_client source
  • A new amqp source and sink that supports AMQP 0.9.1 (used by RabbitMQ)
Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • Vector fails to load configurations using environment variables outside of string values for configuration options. Fixed in 0.25.1.
  • Vector fails to load multi-file configurations using the global timezone configuration option. Fixed in 0.25.1.
  • The prometheus_remote_write sink doesn’t support specifying the configuration auth.bearer as it should. Fixed in 0.25.1.
  • The abort VRL function emits ERROR, rather than DEBUG, logs when discarding an event. Fixed in 0.25.2.
  • The azure_blob sink incorrectly passes a redacted value for the connection_string, rather than the actual contents. Fixed in 0.25.2.

Vector Changelog

13 new features

  • A chunks function was added to VRL to enable breaking up a text field into multiple chunks of equal or lesser length.
    Thanks to briankung for contributing this change!
  • A new http_client source has been added. This source makes HTTP requests to the configured endpoint, on an interval, and turns the response into events based on the configured framing and decoding options.

  • Event metadata fields can now be referred to in VRL and configuration options that take event field paths by using the %<field name> syntax. For example, to refer to a metadata field on the event called foo you would use %foo. For now, most metadata fields are user-defined (e.g. %foo ="bar" in VRL), but in the future Vector will add more metadata, like event ingest timestamp.

    As part of this, the metadata functions in VRL (set_metadata_field, get_metadata_field', and remove_metadata_field) have been deprecated. Instead, the new %syntax should be used to access, modify, and remove metadata fields using normal VRL path operations. For example, setting a metadata field offoowould look like%foo = “bar”`. Please see the upgrade guide for more details on the deprecation.

  • The prometheus_remote_write sink can now be used with Amazon Managed Service for Prometheus by using AWS request signing.
    Thanks to notchairmk for contributing this change!
  • The internal log rate limiting that Vector does to avoid flooding its output during catastrophic events is now configurable via --internal-log-rate-limit on the CLI or the VECTOR_INTERNAL_LOG_RATE_LIMIT environment variable. The default is 10 seconds.
  • A new amqp source and sink have been added to receive or send data via the AMQP 0.9.1 protocol, including RabbitMQ.
    Thanks to dbcfd for contributing this change!
  • The pulsar sink now supports configuring which field name to use as the partition key in Pulsar via partition_key_field.
    Thanks to miton18 for contributing this change!
  • The prometheus_exporter sink now supports configuring HTTP basic auth credentials to restrict access.
    Thanks to zamazan4ik for contributing this change!
  • The parse_xml VRL function avoids a panic due to certain invalid XML.
    Thanks to Zettroke for contributing this change!
  • The elasticsearch sink can now be configured to send data to multiple Elasticsearch instances via the new endpoints parameter.
    Thanks to ktff for contributing this change!
  • The aws_s3 source now ignores the s3:TestEvent SQS messages that AWS sends when wiring up S3 bucket notifications. Previously, Vector would error when consuming these events.
    Thanks to bencord0 for contributing this change!
  • A parse_cef VRL function was added for parsing ArcSight Common Event Format (CEF).
    Thanks to ktff for contributing this change!

  • The loki sink now supports sending data to Loki via its native snappy-compressed protobuf protocol by setting compression to snappy. This is the new default, but can be reverted to the previous behavior by setting compression to none.

    Please see the upgrade guide for more details.


    Thanks to xdatcloud for contributing this change!

6 enhancements

  • All components were audited to ensure compliance with the component specification with respect to component_discarded_events_total and component_errors_total internal metrics. Generally this meant adding missing metrics and logs, missing metric and logs labels, and applying internal log rate limits consistently.
  • The file sink has added support for zstandard compressed output.
    Thanks to hdhoang for contributing this change!
  • The parse_key_value function now handles duplicate keys by grouping the values into an array.
    Thanks to ktff for contributing this change!
  • The del function now has an optional compact parameter that can be used to delete the parent of the path being deleted if there are no other fields (for objects) or elements (for arrays) in it.
  • A new build_info internal metric was added to report Vector version and build information. This can be useful for monitoring a fleet of Vector instances.
  • A keys and values function were added to VRL to fetch the keys and values, respectively, of an object, into an array.

11 bug fixes

  • The datadog_metrics sink now correctly aggregates counters emitted for the same timeseries within a single second (the timestamp granularity). Previously, it sent them through as-is to Datadog which processed them as last-write-wins.

  • The internal_metrics source now defaults to setting the host tag as this behavior seems to be less surprising to users. It can be suppressed by setting the host_key option to "".

    Please see the upgrade guide for more details.

  • The lua transform can now load dynamically linked libraries. Previously the needed symbols were being stripped from Vector.
  • The honeycomb sink now uses the correct timestamp field name.
    Thanks to McSick for contributing this change!
  • The clickhouse sink now fills in unprovided URL parts with defaults, for example by prefixing http:// if only an address and port are provided.
    Thanks to zamazan4ik for contributing this change!
  • The syslog source no longer panics when parsing invalid dates, instead it logs an error.
  • Using vector validate with a configuration that includes the stdin source no longer blocks.
  • The mongodb_metrics source no longer requires that the fetched statistics have a record field (used to emit mongod_metrics_record_moves_total). This field is not returned by MongoDB 6.
    Thanks to KernelErr for contributing this change!
  • The file_descriptor and stdin sources no longer assume inputs are always logs so that the source can be used to ingest metrics and traces via the native and native_json codecs.
    Thanks to vimalk78 for contributing this change!
  • VRL now correctly handles assignment to .tags for metrics. Previously it would overwrite instead of assign.
  • Pre-compile the configuration field templates for the log_to_metric transform. This improves performance at runtime.

4 chore

  • The deprecated % operator was removed from VRL in-lieu of the new mod function that was added in v0.24.0.

    Please see the upgrade guide for more details.

  • The long deprecated v1 protocol of the vector source and sink (indicated by specifying version = "1") has finally been removed.

    Please see the upgrade guide for more details.

  • The unused shutdown_timeout_secs option on the vector source was removed. Please see the upgrade guide for more details.
    Thanks to Zettroke for contributing this change!
  • Version 1 of the lua transform API has been officially deprecated and will be removed in a newer version of Vector. Please see the upgrade guide for more details.

Download Version 0.25.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...