logback-kafka-appender

This appender lets your application publish its application logs directly to Apache Kafka.

Logback incompatibility Warning

Due to a breaking change in the Logback Encoder API you need to use at least logback version 1.2.

Full configuration example

Add

logback-kafka-appender

logback-classic

[maven pom.xml]

    com.github.danielwegener
    logback-kafka-appender
    0.2.0
    runtime


    ch.qos.logback
    logback-classic
    1.2.3
    runtime

// [build.sbt]
libraryDependencies += "com.github.danielwegener" % "logback-kafka-appender" % "0.2.0"
libraryDependencies += "ch.qos.logback" % "logback-classic" % "1.2.3"

This is an example

logback.xml

PatternLayout

[src/main/resources/logback.xml]


<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
    </encoder>
</appender>

<!-- This is the kafkaAppender -->
<appender name="kafkaAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
        <topic>logs</topic>
        <keyingstrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy"></keyingstrategy>
        <deliverystrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"></deliverystrategy>

        <!-- Optional parameter to use a fixed partition -->
        <!-- <partition>0</partition> -->

        <!-- Optional parameter to include log timestamps into the kafka message -->
        <!-- <appendTimestamp>true</appendTimestamp> -->

        <!-- each <producerConfig> translates to regular kafka-client config (format: key=value) -->
        <!-- producer configs are documented here: https://kafka.apache.org/documentation.html#newproducerconfigs -->
        <!-- bootstrap.servers is the only mandatory producerConfig -->
        <producerconfig>bootstrap.servers=localhost:9092</producerconfig>

        <!-- this is the fallback appender if kafka is not available. -->
        <appender-ref ref="STDOUT"></appender-ref>
    </appender>

<root level="info">
    <appender-ref ref="kafkaAppender"></appender-ref>
</root>

You may also look at the complete configuration examples

Compatibility

logback-kafka-appender depends on

org.apache.kafka:kafka-clients:1.0.0:jar

The dependency to kafka-clients is not shadowed and may be upgraded to a higher, api compatible, version through dependency overrides.

Delivery strategies

Direct logging over the network is not a trivial thing because it might be much less reliable than the local file system and has a much bigger impact on the application performance if the transport has hiccups.

You need make a essential decision: Is it more important to deliver all logs to the remote Kafka or is it more important to keep the application running smoothly? Either of this decisions allows you to tune this appender for throughput.

| Strategy | Description | |---|---| |

AsynchronousDeliveryStrategy

Kafka Producer

block.on.buffer.full=false

BlockingDeliveryStrategy

linger.ms

Note on Broker outages

The

AsynchronousDeliveryStrategy

metadata.max.age.ms

In any case, if you want to make sure the appender will never block your application, you can wrap the KafkaAppender with logback's own AsyncAppender or, for more control, the LoggingEventAsyncDisruptorAppender from Logstash Logback Encoder.

An example configuration could look like this:

<!-- This is the kafkaAppender -->
<appender name="kafkaAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
<!-- Kafka Appender configuration -->
</appender>

<appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
    <!-- if neverBlock is set to true, the async appender discards messages when its internal queue is full -->
    <neverblock>true</neverblock>  
    <appender-ref ref="kafkaAppender"></appender-ref>
</appender>

<root level="info">
    <appender-ref ref="ASYNC"></appender-ref>
</root>

Custom delivery strategies

You may also roll your own delivery strategy. Just extend

com.github.danielwegener.logback.kafka.delivery.DeliveryStrategy

Fallback-Appender

If, for whatever reason, the kafka-producer decides that it cannot publish a log message, the message could still be logged to a fallback appender (a

ConsoleAppender

Just add your fallback appender(s) as logback

appender-ref

KafkaAppender

logback.xml

appender-ref

Example:

STDOUT

Note that the

AsynchronousDeliveryStrategy

Producer tuning

This appender uses the kafka producer introduced in kafka-0.8.2. It uses the producer default configuration.

You may override any known kafka producer config with an

Name=Value

boostrap.servers

batch.size

compression.type

linger.ms

Serialization

This module supports any

ch.qos.logback.core.encoder.Encoder

ILoggingEvent

IAccessEvent

PatternLayoutEncoder

LogstashEncoxer

Custom Serialization

If you want to write something different than string on your kafka logging topic, you may roll your encoding mechanism. A use case would be to to smaller message sizes and/or better serialization/deserialization performance on the producing or consuming side. Useful formats could be BSON, Avro or others.

To roll your own implementation please refer to the logback documentation. Note that logback-kafka-appender will never call the

headerBytes()

footerBytes()

Your encoder should be type-parameterized for any subtype of the type of event you want to support (typically

ILoggingEvent

public class MyEncoder extends ch.qos.logback.core.encoder.Encoder {/*..*/}

Keying strategies / Partitioning

Kafka's scalability and ordering guarantees heavily rely on the concepts of partitions (more details here). For application logging this means that we need to decide how we want to distribute our log messages over multiple kafka topic partitions. One implication of this decision is how messages are ordered when they are consumed from a arbitrary multi-partition consumer since kafka only provides a guaranteed read order only on each single partition. Another implication is how evenly our log messages are distributed across all available partitions and therefore balanced between multiple brokers.

The order of log messages may or may not be important, depending on the intended consumer-audience (e.g. a logstash indexer will reorder all message by its timestamp anyway).

You can provide a fixed partition for the kafka appender using the

partition

logback-kafka-appender

| Strategy | Description | |---|---| |

NoKeyKeyingStrategy

HostNameKeyingStrategy

ContextNameKeyingStrategy

ILoggingEvents

ThreadNameKeyingStrategy

ILoggingEvents

LoggerNameKeyingStrategy

ILoggingEvents

Custom keying strategies

If none of the above keying strategies satisfies your requirements, you can easily implement your own by implementing a custom

KeyingStrategy

package foo;
import com.github.danielwegener.logback.kafka.keying.KeyingStrategy;

/* This is a valid example but does not really make much sense */
public class LevelKeyingStrategy implements KeyingStrategy {
    @Override
    public byte[] createKey(ILoggingEvent e) {
        return ByteBuffer.allocate(4).putInt(e.getLevel()).array();
    }
}

As most custom logback component, your custom partitioning strategy may also implement the

ch.qos.logback.core.spi.ContextAware

ch.qos.logback.core.spi.LifeCycle

A custom keying strategy may especially become handy when you want to use kafka's log compactation facility.

FAQ

• Q: I want to log to different/multiple topics!
  A: No problem, create an appender for each topic.

License

This project is licensed under the Apache License Version 2.0.