FileBeat directly to ELS or via LogStash?

elasticsearch logstash kibana filebeat
Myles McDonnell picture Myles McDonnell · Oct 5, 2016 · Viewed 7.6k times · Source

We are installing ELS and Kibana for log aggregation/analysis. The first system to use it is greenfield so we output structured logs from the services that make up our system. Given that we don't need to add structure to our logs I was planning on using FileBeat to ship the logs directly to ELS and not use LogStash. Is this a sensible option or does LogStash have value over and above parsing that we might need? If we do use LogStash can I use that to harvest log files or should I still use FileBeat to pump the logs to LogStash?

Answer

Val picture Val · Oct 5, 2016

Logstash is useful if you need to aggregate logs from many servers and apply some common transformations and filtering to your events.

If your log events are already structured and you are ok with indexing them directly, then you can definitely have Filebeat send them directly to ES. If ES goes down (e.g. for maintenance), Filebeat will retry until it can successfully send the events.

Related questions

How to define seperated indexes for different logs in Filebeat/ELK?

I am wondering how to create separated indexes for different logs fetched into logstash (which were later passed onto elasticsearch), so that in kibana, I can define two indexes for them and discover them. In my case, I have a …

Read More
How to retrieve unique count of a field using Kibana + Elastic Search

Is it possible to query for a distinct/unique count of a field using Kibana? I am using elastic search as my backend to Kibana. If so, what is the syntax of the query? Heres a link to the Kibana …

Read More
Logstash date parsing as timestamp using the date filter

Well, after looking around quite a lot, I could not find a solution to my problem, as it "should" work, but obviously doesn't. I'm using on a Ubuntu 14.04 LTS machine Logstash 1.4.2-1-2-2c0f5a1, and I am …

Read More