vector

by timberio

timberio / vector

High-performance, vendor-neutral observability pipelines.

5.5K Stars 341 Forks Last release: 3 months ago (v0.10.0) Mozilla Public License 2.0 3.1K Commits 26 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Components  •  Guides  •  Docs  •  Blog  •  Download v0.10.0


Vector

What is Vector?

Vector is a lightweight, ultra-fast, open-source tool for building observability pipelines. Compared to Logstash and friends, Vector improves throughput by ~10X while significantly reducing CPU and memory usage.

Principles

  • Reliability First. - Built in Rust, Vector's primary design goal is reliability.
  • One Tool. All Data. - One simple tool gets your logs, metrics, and traces (coming soon) from A to B.
  • Single Responsibility. - Vector is a data router, it does not plan to become a distributed processing framework.

Who should use Vector?

  • You SHOULD use Vector to replace Logstash, Fluent*, Telegraf, Beats, or similar tools.
  • You SHOULD use Vector as a daemon or sidecar.
  • You SHOULD use Vector as a Kafka consumer/producer for observability data.
  • You SHOULD use Vector in resource constrained environments (such as devices).
  • You SHOULD NOT use Vector if you need an advanced distributed stream processing framework.
  • You SHOULD NOT use Vector to replace Kafka. Vector is designed to work with Kafka!
  • You SHOULD NOT use Vector for non-observability data such as analytics data.

Community

  • Vector is downloaded over 100,000 times per day.
  • Vector's largest user processes over 10TB daily.
  • Vector is used by multiple fortune 500 companies with stringent production requirements.
  • Vector has over 15 active contributors and growing.

Documentation

About

Setup

Reference

Administration

Resources

Performance

The following performance tests demonstrate baseline performance between common protocols with the exception of the Regex Parsing test.

| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | SplunkUF | SplunkHF | | ---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | TCP to Blackhole | 86mib/s | n/a | 64.4mib/s | 27.7mib/s | 40.6mib/s | n/a | n/a | | File to TCP | 76.7mib/s | 7.8mib/s | 35mib/s | 26.1mib/s | 3.1mib/s | 40.1mib/s | 39mib/s | | Regex Parsing | 13.2mib/s | n/a | 20.5mib/s | 2.6mib/s | 4.6mib/s | n/a | 7.8mib/s | | TCP to HTTP | 26.7mib/s | n/a | 19.6mib/s | <1mib/s | 2.7mib/s | n/a | n/a | | TCP to TCP | 69.9mib/s | 5mib/s | 67.1mib/s | 3.9mib/s | 10mib/s | 70.4mib/s | 7.6mib/s |

To learn more about our performance tests, please see the Vector test harness.

Correctness

The following correctness tests are not exhaustive, but they demonstrate fundamental differences in quality and attention to detail:

| Test | Vector | Filebeat | FluentBit | FluentD | Logstash | Splunk UF | Splunk HF | | ---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | Disk Buffer Persistence | ✅ | ✅ | ❌ | ❌ | ⚠️ | ✅ | ✅ | | File Rotate (create) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | File Rotate (copytruncate) | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | | File Truncation | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | Process (SIGHUP) | ✅ | ❌ | ❌ | ❌ | ⚠️ | ✅ | ✅ | | JSON (wrapped) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |

To learn more about our correctness tests, please see the Vector test harness.

The Little Details

Data Model

Control Flow

Data Processing

Operations

Reliability

  • Memory safety - Vector is built in Rust and is memory safe, avoiding a large class of memory related errors.
  • Decoupled buffer design - Buffers are per-sink; a bad sink won't bring the entire pipeline to a halt.
  • Intelligent retries - A fibonacci backoff algorithm with jitter makes Vector a good citizen during outages.
  • Backpressure & load shedding - Buffers can be configured to provide back-pressure or shed load.
  • Rate-limited internal logging - Vector's internal logging is rate-limited avoiding IO saturation if errors occur.
  • Sink healthchecks - Healthchecks provide startup safety and prevent deploys with bad configuration.
  • Robust disk buffering - Vector uses
    leveldb
    for robust data durability across restarts.

UX

Installation

Run the following in your terminal, then follow the on-screen instructions.

curl --proto '=https' --tlsv1.2 -sSf https://sh.vector.dev | sh

Or use your own preferred method.

Latest Highlights

view all...


Developed with ❤️ by Timber.io - Security Policy - Privacy Policy

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.