Apache Kafka on Docker

This repository holds a build definition and supporting files for building a Docker image to run Kafka in containers. It is published as an Automated Build on Docker Hub, as

ches/kafka

This build intends to provide an operator-friendly Kafka deployment suitable for usage in a production Docker environment:

• It runs one service, no bundled ZooKeeper (for more convenient development, use Docker Compose!).
• Configuration is parameterized, enabling a Kafka cluster to be run from multiple container instances.
• Kafka data and logs can be handled outside the container(s) using volumes.
• JMX is exposed, for Kafka and JVM metrics visibility.

If you find any shortcomings with the build regarding operability, pull requests or feedback via GitHub issues are welcomed.

Usage Quick Start

Here is a minimal-configuration example running the Kafka broker service, then using the container as a client to run the basic producer and consumer example from the Kafka Quick Start:

# A non-default bridge network enables convenient name-to-hostname discovery
$ docker network create kafka-net

$ docker run -d --name zookeeper --network kafka-net zookeeper:3.4
$ docker run -d --name kafka --network kafka-net --env ZOOKEEPER_IP=zookeeper ches/kafka
$ docker run --rm --network kafka-net ches/kafka 
>   kafka-topics.sh --create --topic test --replication-factor 1 --partitions 1 --zookeeper zookeeper:2181
Created topic "test".
In separate terminals:
$ docker run --rm --interactive --network kafka-net ches/kafka 
>   kafka-console-producer.sh --topic test --broker-list kafka:9092

$ docker run --rm --network kafka-net ches/kafka 
>   kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server kafka:9092

Volumes

The container exposes two volumes that you may wish to bind-mount, or process elsewhere with

--volumes-from

• /data

  : Path where Kafka's data is stored (

  log.dirs

  in Kafka configuration)

• /logs

  : Path where Kafka's logs (

  INFO

  level) will be written, via log4j

Ports and Linking

The container publishes two ports:

• 9092

  : Kafka's standard broker communication

• 7203

  : JMX publishing, for e.g. jconsole or VisualVM connection

Kafka requires Apache ZooKeeper. You can satisfy the dependency by simply linking another container that exposes ZooKeeper on its standard port of 2181, as shown in the above example, ensuring that you link using an alias of

zookeeper

Alternatively, you may configure a specific address for Kafka to find ZK. See the Configuration section below.

A more complex local development setup

This example shows more configuration options and assumes that you wish to run a development environment with Kafka ports mapped directly to localhost, for instance if you're writing a producer or consumer and want to avoid rebuilding a container for it to run in as you iterate. This requires that localhost is your Docker host, i.e. your workstation runs Linux. If you're using something like boot2docker, substitute the value of

boot2docker ip

$ mkdir -p kafka-ex/{data,logs} && cd kafka-ex
$ docker run -d --name zookeeper --publish 2181:2181 zookeeper:3.4
$ docker run -d \
    --hostname localhost \
    --name kafka \
    --volume ./data:/data --volume ./logs:/logs \
    --publish 9092:9092 --publish 7203:7203 \
    --env KAFKA_ADVERTISED_HOST_NAME=127.0.0.1 --env ZOOKEEPER_IP=127.0.0.1 \
    ches/kafka

Configuration

Some parameters of Kafka configuration can be set through environment variables when running the container (

docker run -e VAR=value

• KAFKA_BROKER_ID=0

Maps to Kafka's

broker.id

KAFKA_PORT=9092

Maps to Kafka's

port

KAFKA_ADVERTISED_HOST_NAME=

Maps to Kafka's

advertised.host.name

KAFKA_PORT

KAFKA_ADVERTISED_PORT=9092

As above, for the port part of the advertised address. Maps to Kafka's

advertised.port

KAFKA_DEFAULT_REPLICATION_FACTOR=1

Maps to Kafka's

default.replication.factor

KAFKA_NUM_PARTITIONS=1

Maps to Kafka's

num.partitions

KAFKA_AUTO_CREATE_TOPICS_ENABLE=true

Maps to Kafka's

auto.create.topics.enable

KAFKA_INTER_BROKER_PROTOCOL_VERSION

Maps to Kafka's

inter.broker.protocol.version

KAFKA_LOG_MESSAGE_FORMAT_VERSION

Maps to Kafka's

log.message.format.version

KAFKA_LOG_RETENTION_HOURS=168

Maps to Kafka's

log.retention.hours

• KAFKA_MESSAGE_MAX_BYTES

Maps to Kafka's

message.max.bytes

• KAFKA_REPLICA_FETCH_MAX_BYTES

Maps to Kafka's

replica.fetch.max.bytes

• JAVA_RMI_SERVER_HOSTNAME=$KAFKA_ADVERTISED_HOST_NAME

Maps to the

java.rmi.server.hostname

KAFKA_ADVERTISED_HOST_NAME

ZOOKEEPER_IP=

Required if no container is linked with the alias "zookeeper" and publishing port 2181, or not using

ZOOKEEPER_CONNECTION_STRING

zookeeper.connect

ZOOKEEPER_PORT=2181

Used in constructing Kafka's

zookeeper.connect

ZOOKEEPER_CONNECTION_STRING=

Set a string with host:port pairs for connecting to a ZooKeeper Cluster. This setting overrides

ZOOKEEPER_IP

ZOOKEEPER_PORT

ZOOKEEPER_CHROOT

/v0_8_1

ZooKeeper root path used in constructing Kafka's

zookeeper.connect

/

JMX

Remote JMX access can be a bit of a pain to set up. The start script for this container tries to make it as painless as possible, but it's important to understand that if you want to connect a client like VisualVM from outside other Docker containers (e.g. directly from your host OS in development), then you'll need to configure RMI to be addressed as the Docker host IP or hostname. If you have set

KAFKA_ADVERTISED_HOST_NAME

JAVA_RMI_SERVER_HOSTNAME

For example in practice, if your Docker host is VirtualBox run by Docker Machine, a

run

$(docker-machine ip docker-vm):7203

$ docker run -d --name kafka -p 7203:7203 \
    --link zookeeper:zookeeper \
    --env JAVA_RMI_SERVER_HOSTNAME=$(docker-machine ip docker-vm) \
    ches/kafka

Note that it is fussy about port as well—it may not work if the same port number is not used within the container and on the host (any advice for workarounds is welcome).

Finally, please note that by default remote JMX has authentication and SSL turned off (these settings are taken from Kafka's own default start scripts). If you expose the JMX hostname/port from the Docker host in a production environment, you should make make certain that access is locked down appropriately with firewall rules or similar. A more advisable setup in a Docker setting would be to run a metrics collector in another container, and link it to the Kafka container(s).

If you need finer-grained configuration, you can totally control the relevant Java system properties by setting

KAFKA_JMX_OPTS

start.sh

Fork Legacy

This image/repo was originally forked from relateiq/kafka. My original motivations for forking were:

• Change the Kafka binary source to an official Apache artifact. RelateIQ's was on a private S3 bucket, and this opaqueness is not suitable for a publicly-shared image for reasons of trust.
• Changes described in this pull request.

After a period of unresponsiveness from upstream on pull requests and my repo tallying far more downloads on Docker Hub, I have made further updates and changes with the expectation of maintaining independently from here on. This project's changelog file describes these in detail.