Vector v0.15.0 release notes

The Vector team is pleased to announce version 0.15.0!

This release includes a number of new components for collecting and sending data using Vector:

It also contains a number of additional enhancements and bug fixes. Check out the highlights and changelog for more details.

Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

Mapping one log event into multiple

Author avatar for jszwedko Jesse Szwedko (jszwedko)
type: new feature domain: vrl domain: remap transform

Changelog

21 new features

remap Add parse_ruby_hash function 7688 remap Allow strings to be compared 7704 new source Initial dnstap source 6681 new source New fluent source 7548 new sink New datadog_events sink 7678 cli Add support for --config-dir to read configuration from directories 7767 elasticsearch sink Add support for tempalting the bulk action 7805 elasticsearch sink Add first-class support for data streams 7774 new source Add logstash source 7857 elasticsearch sink Reintroduce data stream support 7903 new sink Add datadog_events sink 7678 new sink Add datadog_events sink 7678 new sink New azure_blob sink 6861 new transform Aggregate transform to reduce metric volume maintaining data 7846 elasticsearch sink Allow metrics to be sent 7932 new source Add eventstoredb_metrics source 6663 new sink Add datadog_events sink 7678 new sink New redis sink 7078 cli Add graph subcommand for generating graph in DOT format 8116 vrl Use timezone from configuration when parsing timestamps without a timezone 8089 new sink Add datadog_events sink 7678

52 enhancements

kubernetes_logs source Changed default for glob_minimum_cooldown_ms to 1s 7693 remap Add match_array function 7752 aws_s3 source Add concurrency support 7712 helm platform Kubernetes recommended labels and additional labels 7687 splunk_hec source Add x-forwarded-for header support 7575 vector source vector sink Batch events per request for v2 7706 dedupe transform Remove warn level internal_event 7812 tests Create affinity pod in separate namespace for k8s tests 7831 clickhouse sink Allow enabling input_format_skip_unknown_fields 7847 dev K8s e2e tests script minor improvements 7853 kafka sink Add acknowledgements support 7677 remap Add assert_eq! function 7866 config Allow wildcards anywhere in input identifiers 7873 helm platform Add metricRelabelings for Prometheus PodMonitor 7914 http source Add binary encoding support for http source 7859 file source Add support for acknowledgements 7816 observability Add flag to send tracing spans to Datadog APM 7929 helm platform Configure a HAProxy for load balancing the aggregator 7782 aws_s3 source Send content_md5 when writing objects to work with S3 object locking 7936 remap Add standalone key support for parse_key_value 7907 vrl Add decode_percent and encode_percent functions 7937 vrl Add encode_key_value function 7751 vrl Add support standalone for standalone key encoding in encode_key_value 7951 transforms Add new filter, datadog_search, condition type 7837 file source Add fingerprint.lines option to use multiple lines to calculate fingeprints 7940 vrl Accept standalone key in parse_logfmt + minor related fix 7977 kafka source Add support for acknowledgements 7787 helm platform Add option to deploy Topology-aware service 7950 aws_s3 source Batch SQS deletes to improve throughput 7992 docker_logs source Re-add named pipe support on Windows 7996 kafka sink Disable sending key when no key_field configured 7860 observability Emit component utilization as a metric 8028 security Harden SystemD service 6696 helm platform Option to add additional ports to the service in vector-agent helm chart 8021 helm platform Add maxUnavailable for updateStrategy in vector-agent helm chart 8035 vrl Add ip_aton and ip_ntoa functions 8042 releasing Refine Vector user for debian 8064 vrl Add parse_xml function 8046 vrl Add format_int and parse_int functions 8078 kubernetes_logs source Expose lines option 8091 gcp_stackdriver_logs sink Allow template syntax in labels of monitored resource descriptor 8131 loki sink Allow label keys to be templated 8118 gcp_stackdriver_logs sink Allow template syntax in log_id 8173 aws_ec2_metadata transform Add ec2 metadata to metric event instances 8157 aws_s3 source Improve performance of S3 source 8215 vrl parse_key_value fixes 8106 vrl Add unnest function 7404 search Add page title to Algolia search index 8241 external docs Add Vector config examples to main page of vector.dev 8243 elasticsearch sink Add support for acknowledgements 8037 helm platform Introduce customConfig key 8168 internal_metrics source Add host tag to internal_metrics 8164

30 bug fixes

codecs Ensure that flat dotted paths are removed by only_fields 7588 remap Fix type handling for and/or expressions that can error 7848 remap Parse nginx logs with and without referrer 7852 new_relic_logs sink Default encoding field to json 7870 sample transform Accept remap conditions 7864 remap handle starting message from nginx 7930 aws_s3 source Fix missing labels for s3 source metrics 7967 observability Add missing tracing context on HTTP client metrics 8007 processing Manage timeouts in multiline better 7944 prometheus_scrape source Don’t crash on out-of-order histograms 8013 remap Fix parse_apache_log to handle lack of thread id 8053 host_metrics source init_roots on build to only output warnings when host_metrics is used 8054 loki sink Avoid flattening objects 8044 external docs Remove job and instance tags from internal_metrics 8076 kafka source add tokio feature to rdkafka to improve performance 8092 dnstap source Add package ‘apt-utils’ in docker image for dnstap integration test 8077 aws service Allow zero size compressed messages 8069 vrl Ensure parse_syslog handles non structured messages 8102 vrl Replace referrer by referer on nginx parser 8086 unit tests check_fields deprecation notice is incorrect 8132 vrl to_int now truncates floats 8145 journald source Fix event metrics reporting 8163 external docs Fix broken README image 8213 metrics Deny adding aggregated summaries 8096 internal_metrics source Add histogram bucket for 64 8161 sinks Fix broken handling of adaptive_concurrency defaults 8189 file source Handle legacy fingerprint checksums from < v0.14.0 8225 file source Write checkpoints before shutdown 8252 observability Fix span labels for http-based sources 8266 elasticsearch sink Fix data stream timestamp handling 8271

141 chore

dev Move ‘Transform’ into core 7701 performance RFC for improving/monitoring build performance 7695 metrics Rework metrics batch buffer 7699 dev Trim unused public functions 7703 Add myself to docs/reference/team.cue 7742 releasing Prep 0.14.0 release 7725 releasing Bump Cargo.toml to 0.15.0 and update install version 7743 deps Bump semver from 1.0.1 to 1.0.2 7716 deps Bump ordered-float from 2.5.0 to 2.5.1 7715 deps Bump maxminddb from 0.17.2 to 0.18.0 7747 tests run k8s-e2e tests on multinode kubernetes 7599 deps Bump semver from 1.0.2 to 1.0.3 7749 tests Fix aws_s3 source integration tests 7761 metrics Make metric data private 7746 deps Bump mongodb from 2.0.0-beta to 2.0.0-beta.1 7754 deps Bump approx from 0.4.0 to 0.5.0 7713 deps Bump assert_cmd from 1.0.4 to 1.0.5 7748 tests Ensure that tracing output is captured 7764 dev Move VectorSink, StreamSink into core 7762 deps Bump security-framework from 2.2.0 to 2.3.0 7770 tests Pin kafka version due to missing zstd support in latest 7779 deps Bump security-framework from 2.3.0 to 2.3.1 7789 deps Bump rust_decimal from 1.14.1 to 1.14.2 7790 ci Warnings break build 7734 observability Log probable healthcheck as debug 7807 ci Add tracking of workflow metrics for common workflows 7811 deps Bump lalrpop-util from 0.19.5 to 0.19.6 7792 deps Bump hyper from 0.14.8 to 0.14.9 7793 deps Bump lalrpop from 0.19.5 to 0.19.6 7817 deps Bump libc from 0.2.95 to 0.2.96 7818 deps Bump itertools from 0.10.0 to 0.10.1 7825 deps Bump crossterm from 0.19.0 to 0.20.0 7826 deps Upgrade curl-sys to 0.4.44+curl-7.77.0 7755 deps Update Cargo.lock files for wasm tests 7780 deps Bump tokio from 1.5.0 to 1.6.1 7685 tests Refactor internal_logs tests to not be flakey 7516 deps Bump anyhow from 1.0.40 to 1.0.41 7835 dev Support older make version 7844 ci Update tested Kubernetes versions 7798 tests Cleanup vector data directory after running k8s tests 7841 ci Use sccache for benchmarks 7781 deps Bump libc from 0.2.96 to 0.2.97 7849 deps Bump rmp-serde from 0.15.4 to 0.15.5 7850 deps Bump env_logger from 0.8.3 to 0.8.4 7851 kubernetes_logs source Adds tests for checkpointing regressions in k8s 6564 performance Initial work on new transform benchmarks 7783 dev Speed up local release builds for testing 7855 dev Add docs about using sccache locally 7861 ci Remove grub installation from Ubuntu bootstraping 7865 ci Start tracking workflow metrics for baseline timings 7869 ci Install rustup when creating docker environment 7871 deps Bump rand_distr from 0.4.0 to 0.4.1 7876 ci Do not login for publish-new-environment on PRs 7888 ci Remove --nocapture from integration tests 7894 deps Bump rand from 0.8.3 to 0.8.4 7877 Build a fat release binary 7895 tests Fix k8s checkpointing test 7897 elasticsearch sink Split up module 7905 ci Fix logstash integration test 7910 deps Bump k8s-openapi from 0.11.0 to 0.12.0 7889 tests Fix flakey pulsar integration test 7918 releasing Move where the DEBUG info is sensed during builds 7924 remap Define type def object and array inner type 7692 config Rename remap to vrl for condition types 7928 deps Bump stream-cancel from 0.8.0 to 0.8.1 7946 deps Bump rust_decimal from 1.14.2 to 1.14.3 7947 tests Fix k8s e2e line assertion 7953 ci Bump Ilshidur/action-discord from 0.3.0 to 0.3.2 7960 ci Force stop sccache before attempting to start it 7962 deps Bump openssl from 0.10.34 to 0.10.35 7959 deps Upgrade to Rust 1.53 7968 deps Bump atomig from 0.2.0 to 0.3.1 7958 deps Bump pulsar from 2.0.1 to 3.0.0 7493 deps Bump trust-dns-proto from 0.20.0 to 0.20.3 7788 deps Bump reqwest from 0.11.3 to 0.11.4 7983 deps Bump tokio-openssl from 0.6.1 to 0.6.2 7985 kubernetes_logs source Revert glob minimum cooldown to 60s 7980 data model Drop impl AtomInteger from status types 7989 deps Bump tokio from 1.6.1 to 1.6.2 7858 dev Flag src/service.rs as Windows-only 7994 deps Bump tokio from 1.6.2 to 1.7.1 7995 deps Bump bollard from 0.10.1 to 0.11.0 7984 ci Bump docker/login-action from 1.9.0 to 1.10.0 8005 tests Glob faster in checkpoint test 8006 file source Add needed shutdown to file rotation test 8010 ci Temporarily ignore checkpoint test 8015 ci Make k8s e2e workflow build faster 8026 dev Get ./scripts/skaffold.sh working on OSX 8022 ci Lower glob_minimum_cooldown_ms for k8s e2e tests 8024 setup Fix config example for windows install 8031 docs Update request defaults in docs 8033 ci Stop ignoring k8s e2e checkpointing test 8032 kubernetes platform Remove checksum/config as it is invalid 8039 ci Add dummy job for cross matrix job 8043 deps Bump maxminddb from 0.18.0 to 0.19.0 8047 deps Bump pulsar from 3.0.0 to 3.0.1 8048 deps Bump once_cell from 1.7.2 to 1.8.0 8049 deps Bump test-case from 1.1.0 to 1.2.0 8050 ci Fix make target for EventStoreDB integration tests 8061 deps Bump mongodb from 2.0.0-beta.1 to 2.0.0-beta.2 8059 deps Bump tracing-subscriber from 0.2.18 to 0.2.19 8051 ci Bump docker/setup-buildx-action from 1.3.0 to 1.4.1 8072 deps Bump indexmap from 1.6.2 to 1.7.0 8084 deps Upgrade rusoto dependencies 8090 tests Resolve example.com instead of vector.dev for tests 8094 deps Bump syslog_loose from 0.10.0 to 0.11.0 8095 kubernetes_logs source Glob uncompressed rotated log files 8040 deps Bump redis from 0.20.0 to 0.20.2 8098 ci Bump docker/build-push-action from 2.5.0 to 2.6.1 8111 deps Bump assert_cmd from 1.0.5 to 1.0.6 8117 ci Bump docker/setup-buildx-action from 1.4.1 to 1.5.0 8122 RFC to update Vector configuration via Helm 8079 deps Bump structopt from 0.3.21 to 0.3.22 8125 deps Bump prettydiff from 0.4.0 to 0.5.0 8126 deps Bump assert_cmd from 1.0.6 to 1.0.7 8127 deps Bump graphql_client from 0.9.0 to 0.10.0 8128 deps Bump tokio from 1.7.1 to 1.8.0 8129 deps Bump thiserror from 1.0.25 to 1.0.26 8144 deps Bump syslog_loose from 0.11.0 to 0.12.0 8143 dev Remove nonexistent aws and gcp teams from CODEOWNERS 8149 external docs New Vector website 7368 deps Bump tokio from 1.8.0 to 1.8.1 8151 deps Bump ordered-float from 2.5.1 to 2.6.0 8152 ci Avoid updating “bundled with” in CI 8177 deps Bump hyper from 0.14.9 to 0.14.10 8169 deps Bump tokio-stream from 0.1.6 to 0.1.7 8170 deps Bump libc from 0.2.97 to 0.2.98 8171 deps Temporarily ignore prost CVE warning 8186 external docs Run Netlify build only on changes to ./docs 8195 docs Add missing newline to tsconfig.json 8196 search Update search indexing logic 8194 deps Bump listenfd from 0.3.3 to 0.3.5 8231 deps Bump nix from 0.21.0 to 0.22.0 8232 deps Bump anyhow from 1.0.41 to 1.0.42 8233 deps Bump maxminddb from 0.19.0 to 0.20.0 8234 search Safelist search widget classes from purging and inject env vars into search.tsx 8198 ci Bump docker/setup-buildx-action from 1.5.0 to 1.5.1 8242 deps Bump tracing-opentelemetry from 0.13.0 to 0.14.0 8235 deps Bump num_enum from 0.5.1 to 0.5.2 8253 deps Bump ordered-float from 2.6.0 to 2.7.0 8254 tests Wait for shutdown in file_start_position_server_restart 8224

What’s next

Enabling Adaptive Concurrency Control by default
We released Adaptive Concurrency Control in version 0.11.0 of Vector, but, up until now, this feature has been opt-in. We’ve been collecting user feedback, making enhancements, and expect to enable this feature as the default in 0.16.0. Users will still be able to configure static concurrency controls as they do now.
End to end acknowledgements

We’ve heard from a number of users that they’d like improved delivery guarantees for events flowing through Vector. We are working on a feature to allow, for components that are able to support it, to only acknowledging data flowing into source components after that data has been sent by any associated sinks. For example, this would avoid acknowledging messages in Kafka until the data in those messages has been sent via all associated sinks.

This release includes support in additional source and sink components that support acknowledgements, but it has not yet been fully documented and tested. We expect to officially release this with 0.16.0.

Kubernetes aggregator role

We are hard at work at expanding the ability to run Vector as an aggregator in Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.

We do expect this to be released with 0.16.0.

Download Version 0.15.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...