Filebeat - parse fields from message line

elasticsearch filebeat
Hadash picture Hadash · Nov 7, 2016 · Viewed 11.4k times · Source

I am using Filebeat to ship log data from my local txt files into Elasticsearch, and I want to add some fields from the message line to the event - like timestamp and log level. For example here is one of my log lines:

2016-09-22 13:51:02,877 INFO 'start myservice service'

My question is: Can I do that by Filebeat -> Elasticsearch or must I go through Logstash?

Answer

A J picture A J · Nov 7, 2016

You can use Filebeat -> Elasticsearch if you make use of the Ingest Node feature in Elasticsearch 5.0. Otherwise, yes, you need to use Logstash.

In both cases you would use a grok filter to parse the message line into structured data. Also you'll want to use a date to parse and normalize the date.

Related questions

Sending json format log to kibana using filebeat, logstash and elasticsearch?

I have logs like this: {"logId":"57aaf6c8d32fb","clientIp":"127.0.0.1","time":"03:11:29 pm","uniqueSubId":"57aaf6c98963b","channelName":"JSPC","apiVersion":"v1","modulName":null,"actionName":"apiRequest","typeOfError":"","statusCode":"","message":"In Auth","exception":"In Auth","logType":"Info"} {"logId":"57aaf6c8d32fb","…

Read More
Running Filebeat in windows

I set up filebeat on windows recently using these instructions https://www.elastic.co/downloads/beats/filebeat but it forces me to keep a cmd prompt open running the command filebeat.exe -c filebeat.yml I am wondering if there …

Read More
Elasticsearch: No handler for type [keyword] declared on field [hostname]

I get above Mapper Parsing Error on Elasticsearch when indexing log from filebeat. I tried both Filebeat -> Elasticserach and Filebeat -> Logstash -> Elasticsearch approach. I have followed their own documentations, I installed filebeat template as …

Read More