Need help with rdkafka-ruby?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

171 Stars 59 Forks MIT License 334 Commits 12 Opened issues


Modern and performant Kafka client library for Ruby based on librdkafka

Services available


Need anything else?

Contributors list


Build Status Gem Version Maintainability


gem is a modern Kafka client library for Ruby based on librdkafka. It wraps the production-ready C client using the ffi gem and targets Kafka 1.0+ and Ruby versions that are under security or active maintenance. We remove Ruby version from our CI builds if they become EOL.

was written because we needed a reliable Ruby client for Kafka that supports modern Kafka at AppSignal. We run it in production on very high traffic systems.

This gem only provides a high-level Kafka consumer. If you are running an older version of Kafka and/or need the legacy simple consumer we suggest using the Hermann gem.

The most important pieces of a Kafka client are implemented. We're working towards feature completeness, you can track that here:


This gem downloads and compiles librdkafka when it is installed. If you have any problems installing the gem please open an issue.


See the documentation for full details on how to use this gem. Two quick examples:

Consuming messages

Subscribe to a topic and get messages. Kafka will automatically spread the available partitions over consumers with the same group id.

config = {
  :"bootstrap.servers" => "localhost:9092",
  :"" => "ruby-test"
consumer =

consumer.each do |message| puts "Message received: #{message}" end

Producing messages

Produce a number of messages, put the delivery handles in an array and wait for them before exiting. This way the messages will be batched and sent to Kafka in an efficient way.

config = {:"bootstrap.servers" => "localhost:9092"}
producer =
delivery_handles = []

100.times do |i| puts "Producing message #{i}" delivery_handles << producer.produce( topic: "ruby-test-topic", payload: "Payload #{i}", key: "Key #{i}" ) end


Note that creating a producer consumes some resources that will not be released until it

is explicitly called, so be sure to call
only as necessary.


A Docker Compose file is included to run Kafka and Zookeeper. To run that:

docker-compose up


cd ext && bundle exec rake && cd ..
to download and compile

You can then run

bundle exec rspec
to run the tests. To see rdkafka debug output:
DEBUG_PRODUCER=true bundle exec rspec
DEBUG_CONSUMER=true bundle exec rspec

After running the tests you can bring the cluster down to start with a clean slate:

docker-compose down


To see everything working run these in separate tabs:

bundle exec rake consume_messages
bundle exec rake produce_messages

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.