Vector v0.49.0 release notes

The Vector team is excited to announce version 0.49.0!

Please refer to the upgrade guide for breaking changes in this release.

Release highlights:

  • A websocket source was introduced. A WebSocket source in Vector enables ingestion of real-time data from services that expose WebSocket APIs.
  • The http sink’s uri and request.headers config fields now support templating, enabling dynamic construction based on event data.
  • The --watch-config flag now also watches for changes in enrichment table files.
  • Fixed a race condition that could cause negative values in the vector_buffer_byte_size and vector_buffer_events gauges.
  • The prometheus_remote_write sink now offers a expire_metrics_secs config option. This fixes an issue where incremental metrics were preserved for the lifetime of Vector’s runtime causing indefinite memory growth.
Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Known issues

  • The protobuf codecs do not support all telemetry types. Specifically, the following applies:

    • Decoder: supports logs.
    • Encoder: supports logs and traces.

    Metrics are not supported. Any future updates will be noted in changelogs.

  • In this version, the aws_s3 default retry_strategy is to never retry. A workaround to restore the correct default behavior is to set:

    retry_strategy:
  type: custom
  status_codes: []

    or alternatively to retry every request:

    retry_strategy:
  type: all

Vector Changelog

11 new features

  • The http sink’s uri config is now templateable, allowing for dynamic URI building based on event fields.
    Thanks to jorgehermo9 for contributing this change!
  • Added a query_settings and async_insert_settings option in the clickhouse sink, which allows users to configure asynchronous inserts.
    Thanks to pm5 for contributing this change!
  • The --watch-config flag now also watches for changes in enrichment table files.
    Thanks to esensar, Quad9DNS for contributing this change!
  • Added time_settings configuration to the dedupe transform, allowing the max_age of items in the deduplication cache to be set. This helps distinguish between true duplicates and expected repetition in data over longer periods of time.
    Thanks to esensar, Quad9DNS for contributing this change!
  • Added a max_packet_size option to set max packet size for the mqtt source and sink.
    Thanks to simplepad for contributing this change!
  • Added a new websocket source.
    Thanks to benjamin-awd for contributing this change!
  • Added the max_size configuration option for memory buffers.
    Thanks to graphcareful for contributing this change!
  • The request_retry_partial behavior for the elasticsearch sink was changed. Now only the failed retriable requests in a bulk is retried (instead of all requests).
    Thanks to Serendo for contributing this change!
    • Added a TTL-based cache for metric sets.
    • Introduced the expire_metrics_secs configuration to the Prometheus remote write sink, leveraging the new TTL-based cache.
    • This resolves an issue where incremental metrics were retained for the entire lifetime of Vector, leading to unbounded memory growth.

    Thanks to GreyLilac09 for contributing this change!
  • Adds support for Redis Sentinel in the redis sink.
    Thanks to 5Dev24 for contributing this change!
  • Adds support for the Redis ZADD command in the redis sink.
    Thanks to 5Dev24 for contributing this change!

6 enhancements

  • Extends retry logic in aws_s3 sink by allowing users to configure which requests to retry.
    Thanks to jchap-pnnl, jhbigler-pnnl for contributing this change!
  • The http sink request.headers configuration now supports templated values.
    Thanks to notchairmk for contributing this change!
  • The enrichment functions now support an optional wildcard parameter where a match succeeds if the field value equals either the wildcard or the actual comparison value.
    Thanks to nzxwang for contributing this change!

  • The nats sink now supports message headers when publishing to JetStream.

    It introduces a configurable, templated Nats-Msg-Id header that ensures a unique ID for each message. This enables broker-level deduplication, resulting in stronger delivery guarantees and exactly-once semantics when combined with idempotent consumers.


    Thanks to benjamin-awd for contributing this change!
  • The protobuf codecs now support all telemetry data types (logs, metrics, traces).
    Thanks to pront for contributing this change!
  • The UnsignedIntTemplate now supports strftime formatting. For example, this %Y%m%d%H template evaluates timestamps to a number.
    Thanks to 5Dev24 for contributing this change!

15 bug fixes

  • The utilization metric is now properly published periodically, even when there are no events flowing through the components.
    Thanks to esensar, Quad9DNS for contributing this change!
  • Fixed a bug in the elasticsearch sink that caused URI credentials to be ignored. They are now correctly used.
    Thanks to ynachi for contributing this change!
  • Previously the postgres sink healthcheck was not implemented correctly. Now Vector can start when healthcheck.enabled is set to false.
    Thanks to jorgehermo9 for contributing this change!
  • Secret names in the configuration can now contain hyphens. For example `“SECRET[systemd.vm-token]” is now valid.
    Thanks to optician for contributing this change!
  • The nats sink now does not return an error when an unresolvable or unavailable URL is provided.Note: If --require-healthy is set, Vector stops on startup.
    Thanks to rdwr-tomers for contributing this change!
  • VRL programs can now read the interval_ms field. This field was previously writeable but not readable.
    Thanks to thomasqueirozb for contributing this change!
  • Fixed an issue where tags were not modifiable via VRL if metrics-tag-values was set to full.
    Thanks to thomasqueirozb for contributing this change!
  • Fixed an issue in the dnstap and tcp sources where throughput could drop significantly when the number of connections exceeded the number of available cores.
    Thanks to esensar, Quad9DNS for contributing this change!
  • Fixed a race condition that caused negative values to be reported by the vector_buffer_byte_size and vector_buffer_events gauges.
    Thanks to vparfonov for contributing this change!
  • Fixed an issue that could cause a panic when reading environment variables containing non-UTF8 data. Such variables are now handled gracefully.
    Thanks to kurochan for contributing this change!
  • Prevented a panic when DD_API_KEY value contains newline characters. Such values are now handled gracefully.
    Thanks to kurochan for contributing this change!
  • Batches encoded using newline-delimited framing now include a trailing newline at the end.
    Thanks to jszwedko for contributing this change!
  • Fixed a potential hang in Vector core caused by the internal concurrent map. This issue could prevent Vector from shutting down cleanly.
    Thanks to jorgehermo9 for contributing this change!
  • Fixed a log_to_metric configuration bug where the all_metrics field could not be used without also specifying metrics. It can now be set independently.
    Thanks to pront for contributing this change!
  • The log_to_metric transforms now emits a pair expansion error. This error was previously silently ignored.
    Thanks to pront for contributing this change!

1 chore

  • Previously heroku_logs and demo_logs sinks could output logs, metrics and traces depending on the decoding. Now they can only output logs. This behavior was unintuitive and undocumented.
    Thanks to thomasqueirozb for contributing this change!

VRL Changelog

0.26.0

Breaking Changes & Upgrade Guide

  • The parse_cef now trims unnecessary whitespace around escaped values in both headers and extension fields, improving accuracy and reliability when dealing with messy input strings. authors: yjagdale (https://github.com/vectordotdev/vrl/pull/1430)

  • The parse_syslog function now treats RFC 3164 structured data items with no parameters (e.g., [exampleSDID@32473]) as part of the main message, rather than parsing them as structured data. Items with parameters (e.g., [exampleSDID@32473 field="value"]) continue to be parsed as structured data. (https://github.com/vectordotdev/vrl/pull/1435)

  • encode_lz4 no longer prepends the uncompressed size by default, improving compatibility with standard LZ4 tools. A new prepend_size flag restores the old behavior if needed. Also, decode_lz4 now also accepts prepend_size and a buf_size option (default: 1MB). authors: jlambatl (https://github.com/vectordotdev/vrl/pull/1447)

New Features

  • Added haversine function for calculating haversine distance and bearing. authors: esensar Quad9DNS (https://github.com/vectordotdev/vrl/pull/1442)

  • Add validate_json_schema function for validating JSON payloads against JSON schema files. An optional configuration parameter ignore_unknown_formats is provided to change how custom formats are handled by the validator. Unknown formats can be silently ignored by setting this to true and validation continues without failing due to those fields. authors: jlambatl (https://github.com/vectordotdev/vrl/pull/1443)

Download Version 0.49.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64 ARM64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...