Use filebeat or logstash to open gzip files

elasticsearch logstash elastic-stack logstash-forwarder lumberjack
Swikrit Khanal picture Swikrit Khanal · Jan 5, 2016 · Viewed 7.4k times · Source

I might be asking for too much here and I think it's just not implemented yet but I have ELK stack setup similar (almost identical) to how this tutorial does it: (https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04) So basically, I have filebeat setup on a client computer, and it sends all the log files in a folder to logstash on the server computer.

What I want to do is be able to have gzip files (with text files in them) and text files a directory, and be able to send them all over to logstash with filebeat. So I need filebeat to send a unziped file, or logstash to be able to look in the gzip file and get the text file inside it.
I looked around the web and this doesn't seem to be implemented yet, but most of what I found was with log forwarder and logstash rather then filebeat, so I'm just hoping maybe there is a way with filebeat.

I might just be being overly hopeful here. If you don't know a method to do this then I'd like to know too.

Answer

ruflin picture ruflin · Jan 7, 2016

This is currently not possible with filebeat. An issue with this feature request was opened here: https://github.com/elastic/beats/issues/637

A potential temporary workaround can be found here: https://discuss.elastic.co/t/backfilling-old-logs-with-filebeat/38428

Related questions

Log storage location ELK stack

I am doing centralized logging using logstash. I am using logstash-forwarder on the shipper node and ELK stack on the collector node.I wanted to know the location where the logs are stored in elasticsearch i didn't see any data …

Read More
Elasticsearch: No handler for type [keyword] declared on field [hostname]

I get above Mapper Parsing Error on Elasticsearch when indexing log from filebeat. I tried both Filebeat -> Elasticserach and Filebeat -> Logstash -> Elasticsearch approach. I have followed their own documentations, I installed filebeat template as …

Read More
How to set an Elasticsearch output template in Logstash

I’m relatively new to Kibana and the ELK (Elasticsearch, Logstash and Kibana) stack and I’ve been doing pretty well setting one up, but I have run into what I see as an odd issue and need some help …

Read More