Why do people ship logs to Logstash with NXLog and not Logstash itself?

logging elasticsearch logstash kibana nxlog
Vagif Abilov picture Vagif Abilov · Sep 5, 2014 · Viewed 9.9k times · Source

I read a blog post about setting up Logstash, Elasticsearch and Kibana, and the author suggested to use NXLog to ship logs from different machines. A typical distributed scenario presented in "The Logstash Book" shows how Logstash can be used in both shipping and indexing role. We are currently experimenting with Logstash and setting up it to ship logs to Elasticsearch was straighforward. So I wonder why people choose NXLog as a log shipper to Logstash instead of using Logstash on both ends.

Answer

rutter picture rutter · Sep 5, 2014

The logstash-forwarder project, formerly known as "Lumberjack", explains it like so:

Resource Usage Concerns

Perceived Problems: Some users view logstash releases as "large" or have a generalized fear of Java.

Actual Problems: Logstash, for right now, runs with a footprint that is not friendly to underprovisioned systems such as EC2 micro instances; on other systems it is fine. This project will exist until that is resolved.

Transport Problems

Few log transport mechanisms provide security, low latency, and reliability.

The lumberjack protocol used by this project exists to provide a network protocol for transmission that is secure, low latency, low resource usage, and reliable.

A Logstash shipper instance isn't especially heavyweight, but if your machine only has 1-2GB of RAM it can be hard to casually allocate several hundred MB to another JVM instance.

Another consideration: what if your shipper nodes are running an OS that isn't supported by Logstash? Logstash can run on Windows, now, but it's still on the buggy side of things. I can't specifically vouch for NXLog in that area, but I gather it's a popular choice.

Related questions

Import JSON Files into Logstash + Elasticsearch + Kibana

So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc... Now, I've been toying …

Read More
How to handle multiple heterogeneous inputs with Logstash?

Let's say you have 2 very different types of logs such as technical and business logs and you want: raw technical logs be routed towards a graylog2 server using a gelf output, json business logs be stored into an elasticsearch cluster …

Read More
Which Serilog sink to use for sending to Logstash?

We started using Serilog in combination with Elasticsearch, and it's a very efficient way to store structure log data (and later visualize them using tools like Kibana). However, I see the advantage of not writing log data directly to the …

Read More