Fluentd to Logstash output plugin

logging logstash log-shipping fluentd
user3195649 picture user3195649 · Apr 18, 2014 · Viewed 10.7k times · Source

I am trying to read from the scribe server using flunetd and output those logs to be stored in logstash for now. I know it's very stupid to log the scribe_central logs to another central logger, but we need this to be done in our current architecture.

Does anyone know if there is any plugin to do that? I searched Google but could not find any.

Answer

Kiyoto Tamura picture Kiyoto Tamura · Apr 19, 2014

For Scribe <-> Fluentd, there is fluent-plugin-scribe:

For Fluentd <-> Logstash, a couple of options:

  1. Use Redis in the middle, and use fluent-plugin-redis and input_redis on Logstash's side. So it would be Fluentd -> Redis -> Logstash. This is what Logstash recommends anyway with log shippers + Logstash.
  2. Alternatively, you can use Fluentd's out_forward plugin with Logstash's TCP input. Logstash has Fluentd codec to handle the input coming from Fluentd.

Related questions

How to debug the logstash file plugin

Is it possible to log actions of the logstash file plugin? (i.e. what files it tries to send, what errors happen, etc)

Read More
How to handle multiple heterogeneous inputs with Logstash?

Let's say you have 2 very different types of logs such as technical and business logs and you want: raw technical logs be routed towards a graylog2 server using a gelf output, json business logs be stored into an elasticsearch cluster …

Read More
Import JSON Files into Logstash + Elasticsearch + Kibana

So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc... Now, I've been toying …

Read More