Vector v0.16.0 release notes

Note: this release contains a regression for disk buffers. If you are using disk buffers, please install 0.16.1 instead.

The Vector team is pleased to announce version 0.16.0!

Be sure to check out the upgrade guide for breaking changes in this release.

This release includes:

It also contains a number of additional enhancements and bug fixes. Check out the highlights and changelog for more details.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Changelog

9 new features

host_metrics source Add disk_used_ratio to host metrics 8317 networking HTTP proxy support 8205 new transform new compound transform 8457 new source Initial NATS source 8112 splunk source Support multiple tokens in splunk_hec source 7552 vrl Support DataDog grok parser - parsing DD grok patterns 8736 pipelines add ComponentId struct 8706 observability Datadog observability for Vector pipelines 8791 external docs Helm chart docs updates - reorging and adding Aggregator docs 8824

42 enhancements

external docs add batch.max_bytes in loki sink 8318 dnstap source integration tests Make sure ITs are independent of container 8347 unit tests Improve diagnostics for unit tests 8166 journald source Add support for journal_files_dir option 8319 dnstap source integration tests Make sure ITs are independent of container 8347 unit tests Improve diagnostics for unit tests 8166 lua transform Switch from rlua to mlua 8055 internal_metrics source Add pid as tag 8387 search website Add page tags to search index 8261 observability Name vector worker threads 8389 performance batch events out metric in Pipeline 8383 vrl Add datadog_match_query; replace filter strategy 8370 internal_logs source Add host and pid fields 8388 external docs Provide per-component config examples 8274 internal_metrics source Allow disabling of the host/pid keys 8444 template website Scrub SVGs using SVG OMG 8443 internal_logs source Allow host_key and pid_key to be configured 8440 kubernetes_logs source New namespace_metadata_annotator 8287 vrl Add parse_user_agent function 8262 windows platform add --stop-timeout command-line argument for restart service sub-command 8523 helm platform extra args for vector container 8556 external docs Add component description to component template 8533 css website Update documentation aesthetic 8416 external docs Add nested config examples 8587 sinks Add support for acknowledgements to more HTTP and batch sinks 8426 internal docs Provide more extensive metadocs for vector.dev 8585 aws_sqs sink Add message_deduplication_id support 8607 helm platform Add DNS Policy and Config, Liveness and Readiness probes. 8633 helm platform Allow creation of Secret resources from Helm charts 8638 external docs Provide links to VRL examples 8697 aws_s3 sink emit processed bytes when sending requests 8692 external docs Provide links to VRL error examples 8698 observability Report more ARC metrics 8721 vector source vector sink prepare for v2 release 8401 remap transform allow providing path to VRL program 8659 observability Custom internal_metrics namespace 8759 loki sink Validate label names 8729 enriching added enrichment_tables section to the config 8524 vrl Add enrichment tables to VRL 8660 enriching added file enrichment table 8730 loki sink Partition HTTP requests to loki by stream & re-enable concurrency 8615 topology Remove ComponentId.value 8808

24 bug fixes

ci Remove Netlify warning about Brewfile 8299 helm platform Fix settings metricRelabelings in Helm chart 8360 docker_logs source Fix retry_backoff_secs 8179 helm platform Fix settings metricRelabelings in Helm chart 8360 docker_logs source Fix retry_backoff_secs 8179 datadog_agent source Fix field naming for agent payloads 8441 vrl parse_xml - handle single element nodes 8451 search website Display title in all search result hierarchies 8465 template website Fix log output template 8514 buffers Actually read 100 events at a time 8541 processing Fix handling ipv6 support for apache error regex client part 8565 vrl update type_def after del 8434 lua transform Adjust memory use metric to be more clear 8622 encoding Ensure that keys for empty collections are not returned by only_fields 8645 loki sink On 404 query top level for healthcheck 8680 observability Do not emit events_in_total with 0 8718 css website Homepage Mobile Adjustments 8760 css website Code block formatting 8786 kubernetes_logs source Fix regex parsing errors 8627 vector sink retry on failed connection 8758 file source Read empty lines 8366 observability add back component_name label 8843 css website Typography Fixes 8860 config config merge also merges proxy 8869

128 chore

Remove sleep from simple_checkpoint 8323 file source Write checkpoints only if they changed 8176 releasing Prepare 0.15.0 release 8247 releasing Update version in Cargo.toml 8349 Fix VRL examples for assert functions 8351 deps bump redis from 0.20.2 to 0.21.0 8358 deps bump sha-1 from 0.9.6 to 0.9.7 8359 sinks Rework the ARC test harness to avoid the polling loop 8355 deps bump env_logger from 0.8.4 to 0.9.0 8307 datadog service Rename datadog_logs source to datadog_agent 8350 update to latest version of metrics-tracing-context 8375 releasing Prepare 0.15.0 release 8247 releasing Update version in Cargo.toml 8349 Fix VRL examples for assert functions 8351 deps bump redis from 0.20.2 to 0.21.0 8358 deps bump sha-1 from 0.9.6 to 0.9.7 8359 sinks Rework the ARC test harness to avoid the polling loop 8355 deps bump env_logger from 0.8.4 to 0.9.0 8307 update to latest version of metrics-tracing-context 8375 Remove CODEOWNERS 8372 deps bump tokio from 1.8.1 to 1.8.2 8377 Parse Datadog Log agent payloads faster 8297 merge down v0.15 #8390 8391 releasing Have install.sh script pull from release directory 8393 Teach Event what its size is 8396 deps bump maxminddb from 0.20.0 to 0.21.0 8397 Adjust blackhole sink to use ByteSizeOf trait 8409 deps bump pin-project from 1.0.7 to 1.0.8 8419 Ignore flakey file-source tests 8424 Adjust datadog logs sink defaults 8425 Split sink into multiple files 8427 internal docs Update RFC guidelines and template 8415 datadog_logs sink Strike text encoding from Datadog logs sink 8445 deps bump futures from 0.3.15 to 0.3.16 8453 Break service up into builder/service pattern 8458 deps bump criterion from 0.3.4 to 0.3.5 8466 deps bump hyper from 0.14.10 to 0.14.11 8420 Add scope section to RFC template 8469 merge down website 8476 merge down v0.15 8479 deps bump rust_decimal from 1.14.3 to 1.15.0 8481 Rate limit defaults reconsidered 8472 Small tweaks around the Datadog sink 8488 external docs Move unused CUE sources into backup directory 8490 merge down 0.15 into website 8492 vrl CSV enrichment RFC 8400 deps bump async-trait from 0.1.50 to 0.1.51 8495 deps bump serde_json from 1.0.64 to 1.0.65 8496 deps bump lru from 0.6.5 to 0.6.6 8497 deps bump socket2 from 0.4.0 to 0.4.1 8498 deps bump tokio from 1.8.2 to 1.9.0 8430 deps Upgrade to Rust 1.54 8510 deps bump cached from 0.23.0 to 0.24.0 8519 deps bump serde_json from 1.0.65 to 1.0.66 8520 Update public roadmap URL 8527 Update public roadmap URL 8527 ci bump styfle/cancel-workflow-action from 0.9.0 to 0.9.1 8529 RFC for extracting Helm charts to a separate repo 8452 Root-level docs cleanup 8548 Updating CONTRIBUTING.md with user experience section 8549 deps bump serde from 1.0.126 to 1.0.127 8554 RFC template refinements 8574 deps bump num_enum from 0.5.2 to 0.5.3 8571 Move transform configuration into vector-core 8584 deps bump assert_cmd from 1.0.7 to 1.0.8 8553 deps bump bytesize from 1.0.1 to 1.1.0 8555 Add user experience design document 8576 deps bump semver from 1.0.3 to 1.0.4 8551 deps bump tui from 0.15.0 to 0.16.0 8552 deps bump assert_cmd from 1.0.8 to 2.0.0 8613 deps bump twox-hash from 1.6.0 to 1.6.1 8614 fluent source Treat only with stringy map keys 8612 deps bump mongodb from 2.0.0-beta.2 to 2.0.0-beta.3 8632 deps bump libc from 0.2.98 to 0.2.99 8641 Remove use of repo local charts in k8s-e2e tests 8651 deps bump bitflags from 1.2.1 to 1.3.1 8673 deps bump cached from 0.24.0 to 0.24.1 8681 Separate internal and external docs 8557 releasing Merge 0.15.2 into website 8694 Update documentation for new concurrency, rate limit values 8690 config Multiple pipelines RFC 8378 Enhance request retry warning message 8664 Clarify message when droppping large events 8704 deps bump windows-service from 0.3.1 to 0.4.0 8685 deps bump matches from 0.1.8 to 0.1.9 8711 deps bump httpmock from 0.5.8 to 0.6.2 8715 deps bump cached from 0.24.1 to 0.25.0 8719 deps bump k8s-openapi from 0.12.0 to 0.13.0 8639 docs Update website development branch documentation 8722 Add highlight for new helm.vector.dev repo 8712 vrl resolve local vs CI timezone issues in tests 8646 deps bump anyhow from 1.0.42 to 1.0.43 8733 deps bump nix from 0.22.0 to 0.22.1 8734 vrl add stdlib features check 8658 kubernetes_logs source Document minimum required version for Kubernetes 8744 Remove integration tests duplicated by ct in timberio/helm-charts 8748 observability Only refer to components by id rather than name 8417 host_metrics source Break this source into modules 8755 deps bump num_enum from 0.5.3 to 0.5.4 8757 deps bump rustyline from 8.2.0 to 9.0.0 8768 deps bump tracing-subscriber from 0.2.19 to 0.2.20 8769 deps bump openssl from 0.10.35 to 0.10.36 8774 deps bump rmpv from 0.4.7 to 1.0.0 8775 deps bump tracing-core from 0.1.18 to 0.1.19 8777 deps bump serde_yaml from 0.8.17 to 0.8.18 8778 Migrate API_KEY_MATCHER and log_schema() into DatadogAgentSource 8789 Rename datadog_logs files to datadog_agent 8802 deps Regenerate Cargo.lock 8804 deps bump tokio from 1.9.0 to 1.10.0 8716 ci bump actions/github-script from 4.0.2 to 4.1 8818 ci bump docker/build-push-action from 2.6.1 to 2.7.0 8819 deps bump opentelemetry dependencies 8807 loki sink Add test for unordered retries 8724 deps bump libc from 0.2.99 to 0.2.100 8828 deps bump serde from 1.0.127 to 1.0.128 8829 deps bump arc-swap from 1.3.0 to 1.3.1 8831 deps Update vulnerable dependencies 8836 deps Update base docker images 8840 deps bump crossterm from 0.20.0 to 0.21.0 8853 deps bump serde_yaml from 0.8.18 to 0.8.19 8854 deps bump prettydiff from 0.5.0 to 0.5.1 8855 Add vector-aggregator highlight 8800 deps bump serde from 1.0.128 to 1.0.129 8857 deps Update openssl crate 8859 dev Separate out check-style check 8862 deps bump strip-ansi-escapes from 0.1.0 to 0.1.1 8867 deps bump hyper from 0.14.11 to 0.14.12 8868 releasing prepare v0.16.0 release 8851

What’s next

Enabling Adaptive Concurrency Control by default

We released Adaptive Concurrency Control in version 0.11.0 of Vector, but, up until now, this feature has been opt-in. We’ve been collecting user feedback, making enhancements, and expect to enable this feature as the default in 0.17.0. Users will still be able to configure static concurrency controls as they do now.

This was previously scheduled for 0.16.0 but we wanted to allow more time for testing the feature.

Kubernetes aggregator role

We are hard at work at expanding the ability to run Vector as an aggregator in Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.

We do expect this to be released in 0.17.0.

Download Version 0.16.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...