Tagging the Logs by Logstash - Grok - ElasticSearch

Question

Summary:

I am using Logstash - Grok and elastic search and my main aim is to First accept the logs by logstash, parse them by grok and associate tags with the messages depending on the type of the log, and then finally feed it to the Elastic server to query with Kibana.

I have already written this code but am not able to get the tags in Elastic Search. This is my logstash confif file.

input {
  stdin {
    type => "stdin-type"
  }
}
filter {
  grok {
    tags    => "mytags"
    pattern => "I am a %{USERNAME}"
    add_tag => "mytag"
    named_captures_only => true
  }
}
output {
  stdout { debug => true debug_format => "json"}
  elasticsearch {}
}

Where am I going wrong?

Answer 1

1) I would first start with editing your values to match the data type they represent. For example

     add_tag => "mytag"

actually should have an array as it's value, not a simple string. Change that to

     add_tag => ["mytag"]

as a good start. Double check all your values and verify they are of the correct type for logstash.

2) You are limiting your grok filters to messages that are already tagged with "mytags" based on the config line

    tags => "mytags"

I don't see anywhere where you have added that tag ahead of time. Therefore, none of your messages will even go through your grok filter.

3) Please read the logstash docs carefully. I am rather new to the Logstash/Grok/ES/Kibana etc. world as well, but I have had very similar problems to what you have had, and all of them were solved by paying attention to what the documentation says.