Import JSON Files into Logstash + Elasticsearch + Kibana

Question

So, I have a web platform that prints a JSON file per request containing some log data about that request. I can configure several rules about when should it log stuff, only at certain levels, etc...

Now, I've been toying with the Logstash + Elasticsearch + Kibana3 stack, and I'd love to find a way to see those logs in Kibana. My question is, is there a way to make Logstash import these kind of files, or would I have to write a custom input plugin for it? I've searched around and for what I've seen, plugins are written in Ruby, a language I don't have experience with.

Answer 1

Logstash is a very good tool for processing dynamic files.

Here is the way to import your json file into elasticsearch using logstash:

configuration file:

input 
{
    file 
    {
        path => ["/path/to/json/file"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        exclude => "*.gz"
    }
}

filter 
{
    mutate
    {
        replace => [ "message", "%{message}" ]
        gsub => [ 'message','\n','']
    }
    if [message] =~ /^{.*}$/
    {
        json { source => message }
    }

}

output
{ 
  elasticsearch {
    protocol => "http"
    codec => json
    host => "localhost"
    index => "json"
    embedded => true
  }

    stdout { codec => rubydebug }
}

example of json file:

{"foo":"bar", "bar": "foo"}
{"hello":"world", "goodnight": "moon"}

Note the json need to be in one line. if you want to parse a multiline json file, replace relevant fields in your configuration file:

   input 
{   
    file 
    {
        codec => multiline
        {
            pattern => '^\{'
            negate => true
            what => previous                
        }
        path => ["/opt/mount/ELK/json/*.json"]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        exclude => "*.gz"
    }
}

filter 
{
    mutate
    {
        replace => [ "message", "%{message}}" ]
        gsub => [ 'message','\n','']
    }
    if [message] =~ /^{.*}$/ 
    {
        json { source => message }
    }

}