jocko

by travisjeffery

travisjeffery / jocko

Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Nati...

4.0K Stars 291 Forks Last release: almost 3 years ago (0.0.1) MIT License 531 Commits 1 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Jocko

ci gitter codecov

Kafka/distributed commit log service in Go.

Goals of this project:

  • Implement Kafka in Go
  • Protocol compatible with Kafka so Kafka clients and services work with Jocko
  • Make operating simpler
  • Distribute a single binary
  • Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
  • Smarter configuration settings
    • Able to use percentages of disk space for retention policies rather than only bytes and time kept
    • Handling size configs when you change the number of partitions or add topics
  • Learn a lot and have fun

TODO

  • [x] Producing
  • [x] Fetching
  • [x] Partition consensus and distribution
  • [ ] Protocol
    • [x] Produce
    • [x] Fetch
    • [x] Metadata
    • [x] Create Topics
    • [x] Delete Topics
    • [ ] Consumer group [current task]
  • [x] Discovery
  • [ ] API versioning [more API versions to implement]
  • [ ] Replication [first draft done - testing heavily now]

Hiatus Writing Book

I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.

Reading

Project Layout

├── broker        broker subsystem
├── cmd           commands
│   └── jocko     command to run a Jocko broker and manage topics
├── commitlog     low-level commit log implementation
├── examples      examples running/using Jocko
│   ├── cluster   example booting up a 3-broker Jocko cluster
│   └── sarama    example producing/consuming with Sarama
├── protocol      golang implementation of Kafka's protocol
├── prometheus    wrapper around Prometheus' client lib to handle metrics
├── server        API subsystem
└── testutil      test utils
    └── mock      mocks of the various subsystems

Building

Local

  1. Clone Jocko

    $ go get github.com/travisjeffery/jocko
    
  2. Build Jocko

    $ cd $GOPATH/src/github.com/travisjeffery/jocko
    $ make build
    

    (If you see an error about

    dep
    not being found, ensure that
    $GOPATH/bin
    is in your
    PATH
    )

Docker

docker build -t travisjeffery/jocko:latest .

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

License

Jocko is under the MIT license, see the LICENSE file for details.


We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.