Need help with logzoom?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

458 Stars 33 Forks MIT License 100 Commits 9 Opened issues


A lightweight replacement for logstash indexer in Go

Services available


Need anything else?

Contributors list

No Data

LogZoom - A Lumberjack => Logstash indexer in Go

LogZoom is a lightweight, Lumberjack-compliant log indexer based off the fine work of Hailo's Logslam. It accepts the Lumberjack v2 protocol, which is currently supported by Elastic's Filebeat.

It was written with the intention of being a smaller, efficient, and more reliable replacement for logstash and Fluentd.

What does LogZoom do?

Like Logstash, LogZoom receives JSON data from Filebeat via the Lumberjack v2 protocol and inserts the data into different outputs. For example, let's say your application generated a JSON line for every event:

{"@timestamp":"2016-03-31T22:23:14+0000", "url": ""}
{"@timestamp":"2016-03-31T22:25:14+0000", "url": ""}
{"@timestamp":"2016-03-31T22:26:14+0000", "url": ""}

As the diagram shows, you can then run a single process of LogZoom to receive this data and insert to Elasticsearch, S3, etc:

LogZoom Basic Diagram

Unlike Logstash, however, LogZoom does not attempt to manipulate data in any shape or form. JSON data that arrives from Filebeat is directly sent to outputs as-is.

Many users commonly use Logstash by adding a grok filter, "currently the best way in logstash to parse crappy unstructured log data." LogZoom currently does NOT support this use case; it is designed for software applications that generate structured data directly.

For example, if you are trying to use Kibana, a frontend to Elasticsearch, you may need the

field, which Logstash typically inserts for you. With LogZoom, your application must generate this field in each JSON log line. The advantages of using this approach:
  1. LogZoom doesn't have to decode the JSON, insert a new field, and encode the JSON again. Logstash and Fluentd spend a fair amount of CPU time doing this.

  2. The application explicitly defines the semantics of

    . When we used Logstash, we were confused that each record was stamped when the entry was received by the central Logstash process, not when it was generated by the client. This caused great confusion, as we would often see large gaps in data when the data was just marked with the wrong timestamp.

Supported IO


  • Filebeat (Lumberjack V2 Protocol)
  • Redis Message Queue


  • Redis Message Queue
  • TCP Streaming
  • WebSocket Streaming
  • Elasticsearch
  • S3

Getting Started

1. Create config

Create a YAML config file specifying the desired input and outputs. An example config can be found in examples/example.config.yml:

    ssl_crt: /etc/filebeat/filebeat.crt
    ssl_key: /etc/filebeat/filebeat.key
    host: :7201
    host: :7202
      - http://localhost:9200

2. Run the server

$ go build
$ $GOPATH/bin/logzoom -config=examples/example.config.yml
2016/04/07 20:22:50 Starting server
2016/04/07 20:22:50 Starting buffer
2016/04/07 20:22:50 Starting input filebeat
2016/04/07 20:22:50 Starting output tcp
2016/04/07 20:22:50 Starting output websocket
2016/04/07 20:22:50 Starting output elasticsearch
2016/04/07 20:22:50 Setting HTTP timeout to 1m0s
2016/04/07 20:22:50 Setting GZIP enabled: false
2016/04/07 20:22:50 Connected to Elasticsearch

Streaming logs via TCP

nc localhost 7201

Streaming logs via WebSocket

Connect to http://localhost:7202 in a browser.
A list of known sources will be displayed.

Elasticsearch support

Note that currently only Elasticsearch 1.x is supported. If you need 2.x support, I think it is just a matter of updating LogZoom to use Olliver Eilhard's 3.x client.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.