Log management with Graylog & Fluentd

Posted byChristian Moser Posted on16. October 2015 Comments2

A perfect match -Graylog & Fluentd

In the following post, I’ll describe how to quickly setup a docker multi container environment running Graylog and Fluentd. The result is a comprehensive log management platform that is able to collect log data from distributed applications.


Think of Graylog as an open source alternative to Splunk Enterprise, a log management platform for collecting, indexing, and analyzing both structured and unstructured data . Furthermore, you can configure email alerts for certain events and dashboards to monitor your applications, quickly.


Fluentd is an open source data collector, an unified logging layer. It decouples data sources from backend systems by providing a unified logging layer in between. If your applications running within Docker containers, you might be interested in the OOTB logging driver for Fluentd.

Why do I need this?

There are a lot of use cases for such a comprehensive log solution. Especially, if log files are getting bigger and are distributed across multiple servers / applications, it can be quite time consuming analysing the logs.

In my case, I’m using a Scala logback Fluentd appender that forwards all log messages of the application to a Fluentd collector running on another server. Each application got it’s own tag like “applicationXY.prod” or “applicationXY.staging“, so we can differenciate the messages later on in Graylog.

The Fluentd collector or “Fluentd to Graylog forwarder” receives and forwards all log messages to Graylog where they got indexed and persisted.

You might wonder why I don’t send the log messages directly to Graylog? Fluentd has many advantages in terms of log  message handling. For example,  Fluentd supports log file or  memory buffering and failovers to handle situations where Graylog or another Fluentd node would go offline. This is what fluentd is really good at. For more information look at the fluentd out_forward or buffer plugin to get an idea of the capabilities.

Ok, ok.. How do I set this up?!

I put a multi docker container environment together, to speed up this process.

First, it builds a custom Fluentd to Graylog forward container, based on the official Fluentd container and a Graylog (gelf) plugin. Then it links it to the official docker all-in-one container for Graylog, which consists of Elasticsearch, mongodb, nginx and Graylog of course :-).

git clone https://github.com/onmomo/fluentd-gelf-docker.git

I’d recommend going the easy route and install docker-compose.

Then simply execute following command. This will download the all-in-one container, build the Fluentd-gelf forward container and link it all together.

docker-compose up -d

Now, you should be able to access graylog on [container-IP]:9000 with credentials admin:admin. Go to [container-IP]:9000/system/inputs and launch a new Gelf UDP input with the default settings:


Graylog is now listening on port 12201 to receive messages from the Fluentd to Graylog container we built. The container expects messages in the Fluentd format (json) on its TCP input on port 24224 with a tag “gelf.app.XYZ and forwards them to Graylog.

You can now start logging with your Fluentd appender and setting up different streams and dashboards within Graylog by separating the log messages by its “gelf.app.XYZ” tags since they are also forwarded to Graylog.

For more detailed instructions, please check out the Readme.

Fluentd log appenders

Since this is application dependent, here are some links that might help.








2 People reacted on this

  1. Hi,

    thanks for the post. I wonder what is the exact benefit of using fluentd over setting gelf as the logging driver for every container and point it to the Graylog instance.

    I’m currently searching for an option that would not be reliant on the Graylog being up since my containers won’t start with Graylog down. I would best like some logging forwarder that is able to buffer/store to file log messages when the destination system is not available and then try to catch up.

    Best regards,

    Ondrej Burkert

    1. You’re welcome.
      You just answered your question. 🙂 You could use one or multiple fluentd forwarder with failover and file buffering configured to handle these situations. This is what fluentd is really good at. Collecting and forwarding messages from various sources with integrated failover and buffering. For more information look at the fluentd out_forward plugin to learn about its capabilities.

      Regards Christian

Comments are closed.