Vector v0.37.0 release notes

The Vector team is pleased to announce version 0.37.0!

Be sure to check out the upgrade guide for breaking changes in this release.

In addition to the usual enhancements and bug fixes, this release also includes

  • ARMv6 builds of Vector including Debian archives and container images. The Debian archives, for now, are only hosted as release assets and are not published to the Debian repository. We are looking into publishing them there in the future. No RPM packages are built at this time. Kudos to @wtaylor for this contribution.
  • A new mqtt sink to emit events from Vector using the MQTT protocol. A source is in the works. Kudos to the contributors that pushed this forward: @astro, @zamazan4ik, and @mladedav.
  • The dnstap source now supports reading events over TCP. Kudos to @esensar for this contribution.
  • A new mmdb enrichment table type for loading arbitrary mmdb databases and not just GeoIP ones. Kudos to @esensar for this contribution.
  • A new pulsar source for receiving events from Pulsar. Kudos to @zamazan4ik and @WarmSnowy for this contribution.
Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • The geoip2 enrichment table type stopped handling GeoLite2-City mmdb types. This is fixed in v0.37.1.
  • The parse_ddtags setting added to the datadog_agent source incorrectly parses the tags into an object instead of an array. The datadog_logs sink also fails to reconstruct the parsed tags. This will be fixed in v0.38.0.

Vector Changelog

6 new features

  • Vector can send logs to a MQTT broker through the new mqtt sink.
    Thanks to astro, zamazan4ik, StephenWakely, mladedav for contributing this change!
  • Added support for parsing EDNS EDE (Extended DNS errors) options
    Thanks to esensar, Quad9DNS for contributing this change!
  • Added lowercase_hostnames option to dnstap source, to filter hostnames in DNS records and lowercase them for consistency.
    Thanks to esensar, Quad9DNS for contributing this change!
  • Added support for permit_origin config option for all sources with TCP mode (fluent, logstash, statsd, syslog).
    Thanks to esensar, Quad9DNS for contributing this change!
  • Added support for custom MMDB enrichment tables. GeoIP enrichment tables will no longer fall back to City type for unknown types and will instead return an error. New MMDB enrichment table should be used for such types.
    Thanks to esensar, Quad9DNS for contributing this change!
  • A new source has been added that can receive logs from Apache Pulsar.
    Thanks to zamazan4ik, WarmSnowy for contributing this change!

8 enhancements

  • ARMv6 builds are now provided as binaries, .deb archives and container images (alpine and debian).
    Thanks to wtaylor for contributing this change!
  • A new configuration option rotate_wait_secs was added to the file and kubernetes_logs sources. rotate_wait_secs determines for how long Vector keeps trying to read from a log file that has been deleted. Once that time span has expired, Vector stops reading from and closes the file descriptor of the deleted file, thus allowing the OS to reclaim the storage space occupied by the file.
    Thanks to syedriko for contributing this change!
  • A new EXPRESS_ONEZONE option was added to storage_class for aws_s3 sink.
    Thanks to siavashs for contributing this change!
  • Added support for more DNS record types (HINFO, CSYNC, OPT, DNSSEC CDS, DNSSEC CDNSKEY, DNSSEC KEY)
    Thanks to esensar, Quad9DNS for contributing this change!
  • Improves TLS support for greptimedb sink. tls.ca_file is no longer required for enabling TLS. Just use tls = {} in toml configuration when your server is hosting a public CA.
    Thanks to sunng87 for contributing this change!

  • The datadog_agent source now contains a configuration setting parse_ddtags, which is disabled by default.

    When enabled, the ddtags field (a comma separated list of key-value strings) is parsed and expanded into an object in the event.

  • A new configuration option include_paths_glob_patterns has been introduced in the Kubernetes Logs source. This option works alongside the existing exclude_paths_glob_patterns to help narrow down the selection of logs to be considered. include_paths_glob_patterns is evaluated before exclude_paths_glob_patterns.
    Thanks to syedriko for contributing this change!
  • The remap component no longer filters out the file contents from error messages when the VRL program is passed in via the file option.

7 bug fixes

  • Fixed gzip and zlib compression performance degradation introduced in v0.34.0.
    Thanks to Hexta for contributing this change!

  • The datadog_agent source now correctly calculates the value for the metric component_received_event_bytes_total before enriching the event with Vector metadata.

    The source also now adheres to the Component Specification by incrementing component_errors_total when a request succeeded in decompression but JSON parsing failed.

  • The datadog_logs sink no longer requires a semantic meaning input definition for message and timestamp fields.

    While the Datadog logs intake does handle these fields if they are present, they aren’t required.

    The only impact is that configurations which enable the Log Namespace feature and use a Source input to this sink which does not itself define a semantic meaning for message and timestamp, no longer need to manually set the semantic meaning for these two fields through a remap transform.

    Existing configurations that utilize the Legacy namespace are unaffected, as are configurations using the Vector namespace where the input source has defined the message and timestamp semantic meanings.

  • An error log for the Elasticsearch sink that logs out the response body when errors occur. This was a log that used to exist in Vector v0.24.0, but was removed in v0.25.0. Some users were depending on this log to count the number of errors so it was re-added.
  • The fingerprint.ignored_header_bytes option on the file source now has a default of 0.
  • The splunk_hec_logs sink when configured with the raw endpoint target, was removing the timestamp from the event. This was due to a bug in the handling of the auto_extract_timestamp configuration option, which is only supposed to apply to the event endpoint target.

  • We now correctly calculate the estimated JSON size in bytes for the metric component_received_event_bytes_total for the splunk_hec source.

    Previously this was being calculated after event enrichment. It is now calculated before enrichment, for both raw and event endpoints.

3 chore

  • Added support for TCP mode for DNSTAP source. As the dnstap source now supports multiple socket types, you will need to update your configuration to specify which type - either mode: unix for the existing unix sockets mode or mode: tcp for the new tcp mode.
    Thanks to esensar, Quad9DNS for contributing this change!
  • When end-to-end acknowledgments are enabled, outgoing requests in the ClickHouse sink that encounter 500-level errors will now correctly report an errored (retriable) status, rather than a rejected (permanent) status, to Vector’s clients.
  • The default of --strict-env-vars has been changed to true. This option has been deprecated. In a future version it will be removed and Vector will have the behavior it currently has when set to true which is that missing environment variables will cause Vector to fail to start up with an error instead of a warning. Set --strict-env-vars=false to opt into deprecated behavior.

Download Version 0.37.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...