Need help with fluentd-kubernetes-daemonset?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

fluent
884 Stars 765 Forks Apache License 2.0 675 Commits 67 Opened issues

Description

Fluentd daemonset for Kubernetes and it Docker image

Services available

!
?

Need anything else?

Contributors list

Fluentd Daemonset for Kubernetes

Docker Stars Docker Pulls ImageLayers Size ImageLayers Layers

Supported tags and respective
Dockerfile
links

See also dockerhub tags page: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/tags

Debian

Current stable

x86_64 images
  • Azureblob
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-azureblob-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-azureblob-1
  • Elasticsearch7
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-elasticsearch7-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-elasticsearch7-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
  • Elasticsearch6
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-elasticsearch6-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-elasticsearch6-1
  • Loggly
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-loggly-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-loggly-1
  • Logentries
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-logentries-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-logentries-1
  • Cloudwatch
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-cloudwatch-1.3
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-cloudwatch-1
  • Stackdriver
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-stackdriver-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-stackdriver-1
  • S3
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-s3-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-s3-1
  • Syslog
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-syslog-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-syslog-1
  • Forward
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-forward-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-forward-1
  • Gcs
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-gcs-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-gcs-1
  • Graylog
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-graylog-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-graylog-1
  • Papertrail
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-papertrail-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-papertrail-1
  • Logzio
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-logzio-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-logzio-1
  • Kafka
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-kafka-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-kafka-1
  • Kafka2
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-kafka2-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-kafka2-1
  • Kinesis
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-kinesis-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-kinesis-1
arm64 images
  • Azureblob
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-azureblob-arm64-1.3
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-azureblob-arm64-1
  • Elasticsearch7
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-elasticsearch7-arm64-1.4
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-elasticsearch7-arm64-1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch-arm64
  • Elasticsearch6
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-elasticsearch6-arm64-1.1
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-elasticsearch6-arm64-1
  • Loggly
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-loggly-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-loggly-arm64-1
  • Logentries
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-logentries-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-logentries-arm64-1
  • Cloudwatch
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-cloudwatch-arm64-1.3
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-cloudwatch-arm64-1
  • Stackdriver
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-stackdriver-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-stackdriver-arm64-1
  • S3
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-s3-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-s3-arm64-1
  • Syslog
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-syslog-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-syslog-arm64-1
  • Forward
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-forward-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-forward-arm64-1
  • Gcs
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-gcs-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-gcs-arm64-1
  • Graylog
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-graylog-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-graylog-arm64-1
  • Papertrail
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-papertrail-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-papertrail-arm64-1
  • Logzio
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-logzio-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-logzio-arm64-1
  • Kafka
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-kafka-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-kafka-arm64-1
  • Kafka2
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-kafka2-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-kafka2-arm64-1
  • Kinesis
    Dockerfile
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12.2-debian-kinesis-arm64-1.0
    • docker pull fluent/fluentd-kubernetes-daemonset:v1.12-debian-kinesis-arm64-1

You can also use

v1-debian-PLUGIN
tag to refer latest v1 image, e.g.
v1-debian-elasticsearch
. On production, strict tag is better to avoid unexpected update.

See dockerhub's tags page for older tags.

Old stable

v0.12 development has been ended. These images are never updated.

Alpine Linux (This is deprecated. Use Debian images instead)

What is Fluentd?

Fluentd Logo

Fluentd is an open source data collector, which lets you unify the data collection and consumption for a better use and understanding of data.

www.fluentd.org

Image versions

Fluentd versioning is as follows:

| Series | Description | |--------|-------------------------------------| | v1.x | current stable | | v0.12 | Old stable, no longer updated |

Settings

Default image version

Default YAML uses latest v1 images like

fluent/fluentd-kubernetes-daemonset:v1-debian-kafka
. If you want to avoid unexpected image update, specify exact version for
image
like
fluent/fluentd-kubernetes-daemonset:v1.8.0-debian-kafka-1.0
.

Run as root

This is for v0.12 images.

In Kubernetes and default setting, fluentd needs root permission to read logs in

/var/log
and write
pos_file
to
/var/log
. To avoid permission error, you need to set
FLUENT_UID
environment variable to
0
in your Kubernetes configuration.

Use your configuration

These images have default configuration and support some environment variables for parameters but it sometimes doesn't fit your case. If you want to use your configuration, use ConfigMap feature.

Each image has following configurations:

  • fluent.conf: Destination setting, Elaticsearch, kafka and etc.
  • kubernetes.conf: k8s specific setting.
    tail
    input for log files and
    kubernetes_metadata
    filter
  • tailcontainerparse.conf: parser setting for
    /var/log/containers/*.log
    . See also "Use CRI parser for containerd/cri-o" logs section
  • prometheus.conf: prometheus plugin for fluentd monitoring
  • systemd.conf: systemd plugin for collecting systemd-journal log. See also "Disable systemd input" section.

Overwrite conf file via ConfigMap. See also several examples:

Use CRI parser for containerd/cri-o logs

This feature is available since v1.12.0-xxx-1.1.

By default, these images use

json
parser for
/var/log/containers/
files because docker generates json formatted logs. On the other hand, containerd/cri-o use different log format. To parse such logs, you need to use
cri
parser
instead.

You can use

cri
parser by overwriting
tail_container_parse.conf
via ConfigMap.
# configuration example

  @type cri

See also CRI parser README

Use FLUENTCONTAINERTAILEXCLUDEPATH to exclude specific container logs

Since v1.9.3 or later images.

You can exclude container logs from

/var/log/containers/
with
FLUENT_CONTAINER_TAIL_EXCLUDE_PATH
. If you have a trouble with specific log, use this envvar, e.g.
["/var/log/containers/logname-*"]
.
  • exclude_path
    parameter document: https://docs.fluentd.org/input/tail#exclude_path
  • Fluentd log issue with backslash: https://github.com/fluent/fluentd/issues/2545

Disable systemd input

If you don't setup systemd in the container, fluentd shows following messages by default configuration.

[warn]: #0 [in_systemd_bootkube] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_kubelet] Systemd::JournalError: No such file or directory retrying in 1s
[warn]: #0 [in_systemd_docker] Systemd::JournalError: No such file or directory retrying in 1s

You can suppress these messages by setting

disable
to
FLUENTD_SYSTEMD_CONF
environment variable in your kubernetes configuration.

Disable prometheus input plugins

By default, latest images launch

prometheus
plugins to monitor fluentd. You can disable prometheus input plugin by setting
disable
to
FLUENTD_PROMETHEUS_CONF
environment variable in your kubernetes configuration.

Disable sed execution on elasticsearch image

This is for older images. Latest elasticsearch images don't use sed.

By historical reason, elasaticsearch image executes

sed
command during startup phase when
FLUENT_ELASTICSEARCH_USER
or
FLUENT_ELASTICSEARCH_PASSWORD
is specified. This sometimes causes a problem with read only mount. To avoid this problem, set "true" to
FLUENT_ELASTICSEARCH_SED_DISABLE
environment variable in your kubernetes configuration.

Running on OpenShift

This daemonset setting mounts

/var/log
as service account
fluentd
so you need to run containers as privileged container. Here is command example:
oc project kube-system
oc create -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml
oc adm policy add-scc-to-user privileged -z fluentd
oc patch ds fluentd -p "spec:
  template:
    spec:
      containers:
      - name: fluentd
        securityContext:
          privileged: true"
oc delete pod -l k8s-app=fluentd-logging

This is from nekop's japanese article.

Note

kafka image doesn't support zookeeper parameters

zookeeper gem doesn't work on Debian 10, so kafka image doesn't include zookeeper gem.

Windows k8s daemonset not supported in this repository

Maintainers don't have k8s experience on Windows. Some users create k8s daemonset on Windows:

  • https://github.com/bgsilvait/k8s-fluentd-windows
  • https://github.com/k1nger/fluentd-windows-daemon

Please check them out.

kafka image suggestion

Using debian-kafka2/debian-kafka2-arm64 images are better than using debian-kafka/debian-kafka-arm64 images. Because debian-kafka2/debian-kafka2-arm64 images use

out_kafka2
plugin but debina-kafka/debian-kafka-arm64 images use deprecated
out_kafka_buffered
plugin.

Maintainers

Some images are contributed by users. If you have a problem/question for following images, ask it to contributors.

  • azureblob : @elsesiy
  • papertrail : @alexouzounis
  • kafka : @erhudy
  • graylog : @rtnpro
  • gcs : @andor-pierdelacabeza
  • Amazon Kinesis : @shiftky
  • logz.io : @SaMnCo / @jamielennox
  • splunkhec: @FutureSharks

Currently, we don't accept new destination request without contribution. See https://github.com/fluent/fluentd-kubernetes-daemonset/issues/293

References

Kubernetes Logging with Fluentd

Issues

We can't notice comments in the DockerHub so don't use them for reporting issues or asking question.

If you have any problems with or questions about this image, please contact us through a GitHub issue.

Pull Request

Update

templates
files instead of
docker-image
files.
docker-image
files are automatically generated from
templates
.

Note: This file is generated from templates/README.md.erb

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.