Vector v0.11.0 release notes

Code name Kubernetes Pipes

The Vector team is pleased to announce 0.11.0. This release is focused on Kubernetes. In addition to the new, first-class Kubernetes integration, we launched a number of complementary features that make Vector a prime candidate for collecting and processing your Kubernetes logs. Check out the highlights and changelog for more details.
Upgrading Vector
When upgrading, we recommend stepping through minor versions as these can each contain breaking changes while Vector is pre-1.0. These breaking changes are noted in their respective upgrade guides.

Highlights

New metrics integrations

Author avatar for jamtur01 James Turnbull (jamtur01)
type: new feature domain: metrics domain: sources
Adaptive Request Concurrency (ARC)

Author avatar for lucperkins Luc Perkins (lucperkins)
type: new feature domain: performance domain: reliability

Changelog

54 new features

new source Initial kubernetes_logs implementation 2653 sinks networking Automatic concurrency management 3094 cli Allow configuring flags via env vars 2149 geoip transform ASN/ISP database support 3265 new transform New remap transform 3341 new source Initial apache_metrics source implementation 3704 observability New GraphQL API 3708 observability add_fields transform Emit overwrite logs as debug (#3803) 4088 windows platform add a way to register and run vector as a service 2896 new sink Initial azure_monitor_logs sink 2811 new source Add host_metrics source 4157 new transform Add metric_to_log transform 3552 new source New aws_firehose source and aws_cloudwatch_logs_subscription_parser transform 4101 new sink New sematext_metrics sink 3501 logdna sink Allow inclusion of env field in request and default_env in config 4077 observability transforms Update log levels for transforms 4235 azure_monitor_logs sink Add support for dedicated Azure region 4507 observability vector top, v1 4431 new sink New humio_metrics sink. 4531 new source Initial mongodb_metrics implementation 4500 remap transform Add new set of functions 3767 remap transform named function arguments 3927 platforms Add musl x86/aarch64 platforms 3701 remap transform add floor, round and ceil remap functions 4646 remap transform add parse_syslog function 4632 remap transform add split function 4290 remap transform add log function 4640 remap transform add “remap-lang” crate 4695 remap transform add replace and flatten remap functions 4703 observability vector top v1 dashboard 4702 processing add “remap” condition type 4743 remap add “match” remap function 4770 reduce transform add “starts_when” option 4771 remap support else-if conditional expression 4814 remap add variable support 4802 remap transform add replace function 4861 new sink Prometheus remote_write 4856 observability vector top, release candidate 4886 new source aws_s3 source 4779 new transform (key/value) kv parser 3284 new sink New nats sink 3605 new source Initial aws_ecs_metrics source 4698 new sink Initial aws_sqs sink 4675 kubernetes_logs source Add ingestion timestamp 4795 new source Initial nginx_metrics source implementation 5209 remap support resolving program to “any” value 5060 remap support query/assignment of object root 5064 remap transform add parse_grok remap function 4992 remap transform add ip address remap functions 5145 new source New prometheus_remote_write source 5144 remap transform Add exists remap function 5195 remap transform Add compact remap function 5231 remap transform add assert remap function 5073 remap transform Add missing remap functions 5312

159 enhancements

cli Cleanup list command 3099 buffers sinks Upgrade all VecBuffer sinks to allow setting max_bytes 3190 socket source Add max_length to UDP 3236 observability stdin source Instrument “stdin” source 3151 observability journald source Add received and invalid line events 3241 prometheus source Provide error context on parse error 3278 observability kafka source Instrument “kafka” source 3187 humio_logs sink Allow configuration of type field 3300 observability prometheus source Update instrumentation 3317 observability syslog source Update instrumentation 3315 observability http source Add internal events for http source 3264 splunk_hec sink Make sourcetype templatable 3351 observability statsd source Add events 3254 observability docker source Add instrumentation 3345 observability splunk_hec source Add instrumentation 3312 data_dog_metrics sink Add DataDog’s distribution metric 2913 splunk_hec sink Add configuration for source 3327 humio_logs sink Allow configuration of type field (#3300) 3337 humio_logs sink Add source configuration to Humio sink 3328 observability logplex source Add instrumentation 3356 observability file source Add more instrumentation 3417 observability ansi_stripper transform Add instrumentation 3421 observability Add instrumentation 3419 observability Add instrumentation 3418 aws_s3 sink Add additional canned ACLs 3439 codecs console sink Add “text” encoding for metrics 3434 observability file source Even more instrumentation 3436 prometheus source Rewrite parser, improve error handlings 3286 observability startup shutdown Add events for starting, stopping, and reloading 3476 observability Add Heartbeat 3502 file sink compression Add support for gzip compression 3373 file sink Sync all data before finishing 3475 observability json_parser transform Enhance instrumentation 3521 observability socket source Add and unify events 3486 observability regex_parser transform Enhance instrumentation 3523 observability grok_parser transform Add internal events 3553 kubernetes_logs source Add the ability to store pod labels flat 3598 kubernetes_logs source Store pod labels flat by default, remove the switch 3602 file source Add file label 3586 observability Add more main events 3582 observability log_to_metric transform Add internal events 3593 wasm transform Implement some UX improvements for WASM 3490 kubernetes platform Adds new Helm template variable for podsLabels. 3610 docker source Multiline support 3607 observability tag_cardinality_limit transform Add internal events 3577 observability coercer transform Add internal events 3554 http sink Increase rate_limit_num to its maximum 3655 networking Add a new options to control the auto concurrency limiter 3690 observability socket sink Fix TcpEventSent 3720 observability swimlanes transform Add internal events 3730 socket sink Add IPv6 supports 3699 log_to_metric transform Add support for summary metric 3726 observability journald source Enhance checkpoint errors with file name 3782 observability reduce transform Add internal events 3812 observability dedupe transform add internal events 3809 observability tokenizer transform add internal events 3807 influxdb_metrics sink Support summary statistic 3846 datadog_metrics sink Support datadog distribution metric 3725 observability Regularize internal event messages 3850 networking security Enable tls by default for papertrail and datadog_logs sinks 3824 observability elasticsearch sink Improve retry error messages 3861 remap transform Accept more timestamp patterns in to_timestamp 3989 kubernetes_logs source Include container_name in kubernetes_logs 4018 gcp_stackdriver sink Insert timestamp into stackdriver message 3833 observability GraphQL client 3778 remap transform add format_timestamp function 4075 remap transform add contains function 4090 remap transform add slice function 4092 remap transform add tokenize function 4093 kubernetes_logs source Add container_image and pod_node_name annotations 4020 observability socket sink Emit warning on incomplete UDP sent 4034 remap transform add strip_ansi_escape_codes function 4170 remap transform add sha2 function 4188 remap transform add sha3 function 4198 observability json_parser transform add field’s value in warn message when failing to parse 4215 docker platform Added distroless-libc and distroless-static docker container bases 4236 observability Add parse_duration function 4186 prometheus sink Add support for summary metric 4032 auth kubernetes platform aws service Add EKS Web Identity Support 4049 observability Initial GraphQL topology 4191 console sink Improve error handling 4288 remap transform add format_number function 4220 observability Bidirectional source/transform/sink GraphQL types 4383 observability log_to_metric transform Improve error messages 4429 metrics sinks Option to specify quantiles 4412 security networking influxdb_metrics sink add TLS settings to influxdb_logs and influxdb_metrics sinks 4037 influxdb_metrics sink Add a tags configuration options to add user-defined tags 4068 debian platform Add vector user to adm in debian packaging 4544 statsd sink Support all socket types in statsd sink 3557 sinks networking compression Add compression level 3032 kubernetes_logs source Allow using custom selectors 4182 aws service kubernetes platform auth Support assume_role with EKS web identity 4406 reduce transform Rename “identifier_fields” to “group_by” 4580 reduce transform “concat_newline” strategy merges using newline 4579 kubernetes_logs source Advanced container filtering 4586 remap transform Add parse_url function 4428 datadog_logs sink networking Support datadog logs new HTTPS transport 4164 logplex source auth Basic auth support 4385 datadog service Added region configuration parameter 4174 windows platform Correctly handle service restart 4408 statsd source Add support for all socket types 4557 observability internal_metrics source Updated internal metrics names to match standards 4647 observability logfmt_parser transform Add internal events 4481 data model metrics Add namespace to Metric 4701 kubernetes platform Force daemonset to redeploy when configmap is updated 4734 observability Topology added/removed GraphQL subscriptions 4581 observability API host metrics 4652 logplex source annotate logs with query parameters 4733 kubernetes_logs source Expose the performance related parameters 4751 reload Resolve port conflict in sinks 4735 kubernetes platform Add the ability to set container ports at vector-agent Helm chart 4835 aws_ec2_metadata transform Add internal events 4480 cli Adds optional file output to generator 4819 log_to_metric transform Add namespace option 4831 remap display full error chain 4859 logdna sink Support template syntax in hostname and tags field 4884 prometheus source Add TLS and authentication options 4881 gcp_pubsub sink Add configurable endpoint 4873 observability Kind/type for vector top 4928 journald source Restart journalctl on errors, save checkpoint on shutdown 4836 sources make scrape interval configurable 4998 observability Humanized formatting for vector top metrics 4945 observability Batch events processed total 4958 observability Added batch subscriptions for component bytes and errors 5002 observability API batch support + tests 5004 observability API version + hostname queries 5018 kubernetes_logs source Add PodIPs into Pod Metadata events 4887 observability More debug info on more HTTP requests 4999 file source add internal option to ignore missing files 5026 cli Edited a few vector top error messages 5034 remap compile-time program result type checking 4902 remap support enum variants for function arguments 5008 remap use path arguments for del and only_field functions 5015 docker source Renamed docker source to docker_logs 5039 remap expressions no longer return an option 5053 observability Rename version -> versionString in GraphQL schema 5074 remap undefined path or variable return null 5056 observability Add I/O (throughput) columns to vector top 5016 file source Expire checkpoints 5146 kafka source Include kafka metadata as optional keys 5153 sampler transform Add rating by index 4918 elasticsearch sink Support basic-auth credentials in endpoint configuration 5095 api Allow querying transform outputs on transform components 5171 metrics Expose internal metrics cardinality as a internal metric counter 4615 observability Add test for component links 5218 loki sink Allow tenant_id to be templatable on loki sink 5204 remap improve arithmetic type checking 5059 mongodb_metrics source Renamed mongo metrics to new naming standards 4699 observability Add ConnectionOpen gauge 4681 data model Use namespace field in metric sinks 4806 data model Use namespace field in metric sources 4833 shutdown Extend Resource to sources 4996 cli Beautify reports of conflicting Resource usage 5048 observability add _total suffix to events 5098 observability Emit FileOpen in file sink and source 4922 observability Incorrect Log Level Message 5183 config Allow JSON and YAML config formats in addition to TOML 5005 observability Enable TLS subscription connections in vector top 5296 pulsar sink introduce encoding schema and pulsar avro schema 5021

77 bug fixes

prometheus source Allow ‘=’ in tag values 3090 regex_parser transform Correctly assign capture group fields 3164 file source Correct an error with line aggregation in continue_through and halt_before 3262 splunk_hec sink Make sourcetype explicit on Splunk sink 3297 debian platform Remove forwarding to syslog from distributed systemd unit 3427 statsd sink Handle Absolute kind for values other than Gauge. 3510 file source Switch off of broadcast channel 3611 http source Handle panic on HTTP request to HTTPS source 3693 security Handle multiple (intermediate) certificates in PEM files 3687 gcp platform rate_limit_num for GCP pubsub and cloud_storage was wrong 3656 elasticsearch sink allow using excepted fields 3716 gcp_cloud_storage sink Apply encoding rules after key prefix 3770 loki sink Apply encoding rules right before encoding 3775 docker source Add retry delay 3732 sinks apply encoding rules in sinks 3766 reload Recreate trigger 3855 sinks instrument and guard against double poll 4024 socket sink Add reconnect loop to TcpSink 4030 kubernetes_logs source Start reading logs at checkpoint 4043 networking Avoid calling peer_addr 3822 shutdown Fix tokio signal behavior to handle a loop of signals 4095 file source more robust handling of split reads 4089 reload Recompute configuration paths 4094 reload windows platform Wait on Windows after components shutdown 4102 config Do not serialize None in in-flight config 4192 statsd sink Sink should emit event on invalid metric 3636 prometheus sink Ensure server starts immediately 4239 startup Fix 101 exit code on api.enabled = true 4254 humio_logs sink Set host key correctly 4229 apache_metrics source Use configured namespace for up metric 4437 sematext_logs sink Rename sink type and fallback on US region by default 4447 config Merge api::Options 4342 file source Add alias to avoid breaking change 4469 cli Use temporary data directory during validate command 4341 aws_s3 sink Propagate close for Fanout and buffers::disk::Writer 4275 prometheus_source Do not automatically append /metrics 4276 http source Don’t dedot keys 4103 apache_metrics source Sanitize URL when tagging metrics 4410 sinks networking Treat all hyper::Error results as retriable 4575 journald source Handle non-ASCII data in all values 4608 loki sink Send auth header for healthchecks 4604 debian platform Fix flag for usermod in postinst deb script 4694 new_relic_logs sink Fix API max payload size 4707 journald source Fix default-msvc feature 3246 journald source Fix typo in metrics name 3252 config expand macros before validation 3894 windows platform Update tokio signal to avoid unnecessary shutdowns. 4025 azure_monitor_logs sink Use Signer::sign_to_vec to support older OpenSSL versions 4623 statsd sink Improve networking performance 4668 remap transform syslog test should assume local time zone 4711 datadog_logs sink use application/json for json encoding to Datadog 4690 config Make query_parameters optional 4782 syslog Uses IP address as source_ip instead of IP:Port in Syslog source UDP mode 4793 config support numbers in paths 4800 clickhouse sink use custom retry logic 4813 remap unescape forward slash in regex 4864 syslog source Parse RFC3614 messages lacking an app name 4876 blackhole sink Make print_amount an optional option with a default of 1000 4883 humio_metrics sink dont use flatten for logs config fields 4924 wasm transform Update WASM transform to implement TaskTransform 4934 rfcs Fix markdown errors 4954 http source Add span to future 4944 topology Fix fanout remove-while-iterating bug 5027 http sink Fix JSON compression 5050 splunk_hec source Flush messages 5069 topology Flush sinks in a couple of more places 5082 prometheus sink Fix output timestamp 5118 file source clean up file format and fix features 5133 cli Handle config warnings for validate command 5044 aws_s3 sink Fix incorrect _IA storage class names 5245 sinks Add missing inventory::submit 3988 observability make instrumentation only on trace 4765 prometheus sink Prometheus uses cumulative histogram 5001 networking Handle unexpected errors in ARC controller 5267 ci Update release Windows task 5295 ci Update release-docker task 5299 ci Change packages to have full versions and architectures 5303

45 chore

tests Kubernetes E2E test framework 2702 tests tcp merge_and_fork 3340 operations Add build.rs to CI trigger paths 3367 ci E2E CI flows optimization 3353 ci Ship Helm Chart from release CI workflow 3459 sinks Wrap StreamSink around TcpSink 3495 performance Only update auto-concurrency RTT from OK responses 3493 tests Fix Splunk integration test 3661 ci add deny.toml 3765 ci update next_addr_v6 3785 tests Fix journald tests 3794 networking Rework auto concurrency backpressure logic 3804 ci fix clippy warnings for integration tests 4171 networking Adjust auto concurrency tuning defaults 4476 ci Add missing dependency to nightly docker-release 4747 observability Switch from topology to components terminology 4761 blackhole sink Count total bytes for logs 4705 logs Updated remaining log messages to match standards 4737 remap transform swap out remap impl with new remap-lang crate 4709 ci Run tests in on pull requests 4736 platforms Add back armv7 support 4871 config Add Cue sources for the Vector CLI 4880 ci Fix passing –push arg at build-docker.sh 4900 config Add initial setup for metrics docs in Cue 4892 kubernetes platform Remove hardcoded configs in Helm chart 4938 config Document vector top CLI 4907 tests Fix flaky timely_shutdown_docker test 5101 ci add RUSTSEC-2020-0071 to deny.toml 5117 tests Fix flaky configuration_path_recomputed test 5128 host_metrics source Fix test failure for empty metrics 5134 docker_logs source Upgrade bollard to 0.9 to close#4169 5189 ci pip3 installed binaries 3287 tests color in not tty for tests 3324 ci conditional compilation for unix source 3298 tests doc test in event/metric 3461 external docs Fixed language on code fence in CONTRIBUTING.md 3560 ci Remove quiet verbose options as environment variables to stop test-shutdown failing 3559 remove unused imports 3776 tests remove failing remap behavior test 4004 Wrong inventory::submit 4033 tests Add missing use 4475 logdna clippy and prometheus skipped field 4508 Removed a stray Atom that snuck back in 4566 Generate valid config for all components 4636 ci tests fix integration tests list 5293

What’s next

Vector's own observability
We will be working to improve Vector’s own observability with high-quality internal metrics, logs, CLI tools, and dashboards. This will likely be the focus of 0.13.
Kubernetes aggregator role
In addition to Vector’s observability, 0.12 will include support for deploying Vector into the aggregator role within Kubernetes. This will allow you to build end-to-end observability pipelines in Kubernetes with Vector. Distributing processing on the edge, centralizing it with an aggregator, or both. If you are interested in beta testing, please join our chat and let us know.
Vector Remap Language (VRL)

Finally, 0.12 will introduce the new Vector Remap Language. This is a Rust-native syntax designed for safe and efficient data mapping. It’s an ultra-performant middle ground between fundamental transforms like add_fields and a full blown runtime like lua.

If interested, you can beta test via the remap transform and use the Vector Remap Language reference to write scripts.

Download Version 0.11.0

Linux
tar.gz
ARM64 ARMv7 x86_64
Linux (deb)
deb
ARM64 ARMv7 x86_64
Linux (rpm)
rpm
ARM64 ARMv7 x86_64
macOS
tar.gz
x86_64
Windows
zip
x86_64
Windows (MSI)
msi
x86_64
browse all files...