Vector v0.14.0 release notes

The Vector team is pleased to announce version 0.14.0!

This release includes a new exec source that can be used to execute arbitrary programs and use their output to generate events in Vector along with a number of enhancements.

Check out the changelog for more details.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Changelog

6 new features

new source New exec source 6876 remap transform Allow emitting multiple events by assigning . to an array 7267 config Initial [provider] implementation - http 7166 remap Add encode_logfmt function 6985 vector sink vector source Add gRPC-based implementation 7197 remap Add redact function to VRL 7250

24 enhancements

graphql api Expose events_in & events_out metrics 6888 kubernetes_logs source Allow configuration of max_line_bytes 7226 cli Update top to use more accurate events_in_total and events_out_total metrics 7286 kafka source kafka sink Emit internal metrics provided by rdkafka 5874 remap Wdd whitespace option to parse_key_value function 7292 datadog sinks Generic site support 7281 http sink Add support for end-to-end acknowledgements 7265 remap Add parse_klog function 7349 datadog_logs sink Add acknowledgements support 7354 datadog source http source Add acknowledgement support 7325 helm platform Allowing setting command for vector-agent chart 7107 heroku_logs source http source Add end-to-end acknowledgements 7431 sinks Add acknowledgements support to more HTTP logs sinks 7411 helm platform Allow for global imagePullSecrets 7500 datadog_logs sink Use per-API key partitions for datadog logs sink 7394 sources Allow for opting out of acknowledgements 7531 buffers Improve disk buffer compaction performance 7264 remap Add match_any function 7414 vector sink Add acknowledgement support 7625 vector source Add acknowledgement support to v2 protocol 7618 aws_s3 sink Add acknowledgements support 7536 datadog_metrics sink Add acknowledgements support 7535 helm platform additional labels for Prometheus Operator 7474 aws_kinesis_firehose source Add record compression support to aws_kinesis_firehose source 7700

25 bug fixes

remap Escape quotes and backslashes when printing VRL strings 6739 graphql api --no-default-features --features api build failure 7284 splunk_hec source Default host to remote address 7293 cli Enable comma separated lists to be used in configuration environment variables that take a list 7316 observability Correct all *_nanoseconds metrics to _seconds 7373 buffers Work around a LevelDB mechanism mismatch 7398 debian platform Add debhelper include tags to Debian scripts 7363 external docs Document real behaviour of the timestamp field in Syslog source 7443 clickhouse sink Use correct batch status 7452 kubernetes_logs source Expose data_dir configuration 7253 kubernetes platform Fix binfmt exclusion in Helm chart 7332 aws_s3 source Properly handle urlencoded S3 object keys 7470 external docs Fix configure command in installation instructions 7498 codecs Filter object and array fields with only_fields 7468 performance Avoid high memory usage from disk buffers at start-up due to scanning the entire buffer 7522 kubernetes_logs source Kubernetes stream errors should be recoverable 7484 kubernetes_logs source Replace incorrect invocation error with WatchStreamFailed 7534 external docs Fix relevant_when for socket.path config 7587 dev Optimistically add ubuntu to docker group 7583 log_to_metric transform Drop events where field is null 7487 cli include error details in vector vrl output 7581 buffers Avoid compacting small disk buffers 7614 aws_cloudwatch_logs sink Fix healthcheck to allow for log group creation at runtime 7601 buffers Improve disk buffer performance by batching deletes 7615 remap Accept quoted lists in parse_aws_alb_log function 7682

141 chore

releasing Merge down of 0.13.0 release 7194 dev Bump to 0.14.0 7205 deps bump cidr-utils from 0.5.1 to 0.5.3 7211 deps bump async-stream from 0.3.0 to 0.3.1 7209 deps bump syn from 1.0.69 to 1.0.70 7210 deps bump hyper from 0.14.5 to 0.14.6 7212 deps Remove RUSTSEC-2020-0146 from deny.toml 7216 performance Consider dropping jemalloc 6781 ci bump actions/github-script from v3.1 to v4.0.1 7213 Adjust import of crate::Event to crate::event::Event 7200 deps bump hyper from 0.14.6 to 0.14.7 7232 deps bump regex from 1.4.5 to 1.4.6 7233 deps bump async-nats from 0.9.12 to 0.9.14 7234 Run cargo-hack on self-hosted machines 7227 buffers Upgrade DiskBuffer to futures 0.3 7165 splunk_hec source Update to futures 0.3 7225 deps bump rust_decimal from 1.11.1 to 1.12.2 7243 deps bump tokio-postgres from 0.7.1 to 0.7.2 7244 ci bump actions/github-script from v4.0.1 to v4.0.2 7245 deps bump libc from 0.2.93 to 0.2.94 7254 deps bump tower from 0.4.6 to 0.4.7 7260 deps bump syn from 1.0.70 to 1.0.71 7261 deps bump openssl from 0.10.33 to 0.10.34 7270 deps bump pulsar from 2.0.0 to 2.0.1 7271 releasing Update install script to 0.13.1 7275 dev created new lookup library 7087 deps bump rust_decimal from 1.12.2 to 1.12.3 7280 deps bump libz-sys from 1.1.2 to 1.1.3 7278 deps bump async-nats from 0.9.14 to 0.9.16 7279 ci Fix action-discord version for test workflow 7285 tests Simplify cargo test usage 7287 deps bump tracing-core from 0.1.17 to 0.1.18 7294 deps bump tui from 0.14.0 to 0.15.0 7295 deps bump built from 0.4.4 to 0.5.0 7297 deps bump rust_decimal from 1.12.3 to 1.12.4 7301 deps bump bstr from 0.2.15 to 0.2.16 7302 ci Make CI tests run a little more quiet 7269 deps bump tracing-subscriber from 0.2.17 to 0.2.18 7298 deps bump snap from 1.0.4 to 1.0.5 7300 ci Run cargo-flake on the project 7307 ci Introduce more aggressive caching 7306 dev Extract Event into vector-core 7240 deps bump rust_decimal from 1.12.4 to 1.13.0 7320 deps bump ordered-float from 2.1.1 to 2.2.0 7321 ci Flag benchmark runs that fail to actually execute 7330 Reference RFC template in RFC readme 7344 Use ‘cargo hack’ instead of check-component-features 7311 deps bump syn from 1.0.71 to 1.0.72 7334 deps bump regex from 1.4.6 to 1.5.3 7319 deps bump tracing from 0.1.25 to 0.1.26 7296 ci Run tests for all targets on all platforms 7323 sinks Drop unnecessary Request parameter from types 7251 Drop futures01 7343 deps bump sha-1 from 0.9.4 to 0.9.5 7357 deps bump metrics from 0.14.2 to 0.15.1 7313 deps Update grok and dependencies 7176 Drop redundant copy of built info 7365 RFC for emitting multiple log events from remap 7038 Enable stricter clippy lints in vector-core 7341 metrics Remove emits & alias processed_events_total metric 7345 deps bump regex from 1.5.3 to 1.5.4 7374 ci Install llvm on Windows nightly and release builds too 7377 metrics Fix alias for processed_events_total 7378 data model Add metadata support for end-to-end acknowledgements 7141 ci Fix deb verify steps 7385 http sink Distinguish end-to-end errors 7396 deps bump url from 2.2.1 to 2.2.2 7387 Migrate buffers into vector-core 7400 Temporarily revert #7325 7426 ci bump docker/build-push-action from 2 to 2.4.0 7422 ci bump docker/setup-buildx-action from 1 to 1.3.0 7418 ci bump EmbarkStudios/cargo-deny-action from 1 to 1.2.6 7419 ci bump actions/checkout from 2 to 2.3.4 7420 ci bump docker/setup-qemu-action from 1 to 1.1.0 7421 ci bump docker/login-action from 1 to 1.9.0 7430 deps bump serde from 1.0.125 to 1.0.126 7440 deps bump openssl-probe from 0.1.2 to 0.1.4 7415 deps bump futures from 0.3.14 to 0.3.15 7417 deps bump assert_cmd from 1.0.3 to 1.0.4 7439 releasing Ensure latest artifacts are also published with version number 7446 deps bump notify from 4.0.16 to 4.0.17 7450 deps bump sha-1 from 0.9.5 to 0.9.6 7416 influxdb_logs sink Add missing drop in test 7451 ci Install git before checking out for verify jobs 7473 deps bump tokio-test from 0.4.1 to 0.4.2 7475 deps bump tokio-util from 0.6.6 to 0.6.7 7478 deps Upgrade to Rust 1.52.0 7467 deps bump tokio-stream from 0.1.5 to 0.1.6 7479 kubernetes_logs source Add kubeval checks to helm chart updates 6649 swimlanes transform Rename to route 6193 deps Bump Rust version to 1.52.1 7489 deps bump mongodb from 2.0.0-alpha.1 to 2.0.0-beta 7476 dev Configure cross to run correctly in make environment 7466 buffers Make vector-core buffers generic 7471 tests Specify the namespace to run k8s test in 7388 deps bump rustyline from 8.0.0 to 8.1.0 7507 dev Divvy up vector-core 7504 deps bump async-nats from 0.9.16 to 0.9.17 7518 tests Fix broken k8s tests 7517 deps Switch to upstream Registry impl for metrics 7495 deps bump rustyline from 8.1.0 to 8.2.0 7537 dev Pre-install cross in make environment image 7456 RFC for k8s aggregator load balancing 7492 ci Ignore RUSTSEC-2021-0067 while we try to fix it 7559 deps bump thiserror from 1.0.24 to 1.0.25 7554 deps bump cidr-utils from 0.5.3 to 0.5.4 7555 deps bump rust_decimal from 1.13.0 to 1.14.0 7556 deps bump async-stream from 0.3.1 to 0.3.2 7560 deps bump tonic from 0.4.2 to 0.4.3 7562 ci Try building release artifacts on self-hosted runners 7564 ci fix bench-wasm target 7574 deps bump metrics-util from 0.9.0 to 0.9.1 7579 ci parallelize benchmarks 7573 deps Upgrade cue to 0.4.0 7590 ci Try running Linux unit tests with less parallel jobs 7584 dev Reorganize event proto module 7593 deps bump libc from 0.2.94 to 0.2.95 7595 deps bump ordered-float from 2.2.0 to 2.5.0 7596 deps bump hyper from 0.14.7 to 0.14.8 7597 deps bump rust_decimal from 1.14.0 to 1.14.1 7609 deps bump semver from 0.11.0 to 1.0.0 7610 ci bump docker/setup-qemu-action from 1.1.0 to 1.2.0 7611 ci bump docker/build-push-action from 2.4.0 to 2.5.0 7612 dev remove ‘built’ dep and stop needlessly rebuilding on unrelated changes 7602 ci bump actions/cache from 2.1.5 to 2.1.6 7619 ci allow VECTOR_BUILD_DESC to be passed thru during packaging 7620 dev Add changes to Cargo.lock 7621 buffers Move benchmarks into ‘buffers’ sub-crate 7576 Wrap and report batch status receive errors 7635 tests Add e2e k8s test for logs ingestion from Datadog agent to Vector 7628 deps bump tower from 0.4.7 to 0.4.8 7683 ci bump lannonbr/repo-permission-check-action 7686 deps bump nix from 0.20.0 to 0.21.0 7696 deps bump async-nats from 0.9.17 to 0.9.18 7702 deps bump semver from 1.0.0 to 1.0.1 7684 deps bump crc from 1.8.1 to 2.0.0 7627 helm platform Add deprecation notice for rawConfig 7671 releasing Fix commit listing when generating release notes 7721 releasing Remove filters from changelog generation 7722 swimlanes transform Rename to route 6193 config Default end-to-end acknowledgements to off 7720

What’s next

End to end acknowledgements

We’ve heard from a number of users that they’d like improved delivery guarantees for events flowing through Vector. We are working on a feature to allow, for components that are able to support it, to only acknowledging data flowing into source components after that data has been sent by any associated sinks. For example, this would avoid acknowledging messages in Kafka until the data in those messages has been sent via all associated sinks.

This release includes the groundwork for this feature including support in a number of source and sink components that support acknowledgements, but it has not yet been fully documented. We expect to officially release this with 0.15.0.

Kubernetes aggregator role

We are hard at work at expanding the ability to run Vector as an aggregator in Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.

We do expect this to be released in the next few months.

Download Version 0.14.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...