Using Log4J with LogStash

Question

I'm new to LogStash. I have some logs written from a Java application in Log4J. I'm in the process of trying to get those logs into ElasticSearch. For the life of me, I can't seem to get it to work consistently. Currently, I'm using the following logstash configuration:

input {
  file {
    type => "log4j"
    path => "/home/ubuntu/logs/application.log"
  }
}
filter {
  grok {
    type => "log4j"
    add_tag => [ "ApplicationName" ]
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp}  %{LOGLEVEL:level}" ]
  }
}
output {
  elasticsearch {
    protocol => "http"
    codec => "plain"
    host => "[myIpAddress]"
    port => "[myPort]"
  }
}

This configuration seems to be hit or miss. I'm not sure why. For instance, I have two messages. One works, and the other throws a parse failure. Yet, I'm not sure why. Here are the messages and their respective results:

Tags                   Message
------                 -------
["_grokparsefailure"]  2014-04-04 20:14:11,613 TRACE c.g.w.MyJavaClass [pool-2- 
                       thread-6] message was null from https://domain.com/id-1/env-
                       MethodName

["ApplicationName"]    2014-04-04 20:14:11,960 TRACE c.g.w.MyJavaClass [pool-2-
                       thread-4] message was null from https://domain.com/id-1/stable-
                       MethodName

The one with ["ApplicationName"] has my custom fields of timestamp and level. However, the entry with ["_grokparsefailure"] does NOT have my custom fields. The strange piece is, the logs are nearly identical as shown in the message column above. This is really confusing me, yet, I don't know how to figure out what the problem is or how to get beyond it. Does anyone know how how I can use import log4j logs into logstash and get the following fields consistently:

• Log Level
• Timestamp
• Log message
• Machine Name
• Thread

Thank you for any help you can provide. Even if I can just the log level, timestamp, and log message, that would be a HUGE help. I sincerely appreciate it!

Answer 1

I'd recommend using the log4j socket listener for logstash and the log4j socket appender.

Logstash conf:

input {
  log4j {
    mode => server
    host => "0.0.0.0"
    port => [logstash_port]
    type => "log4j"
  }
}
output {
  elasticsearch {
    protocol => "http"
    host => "[myIpAddress]"
    port => "[myPort]"
  }
}

log4j.properties:

log4j.rootLogger=[myAppender]
log4j.appender.[myAppender]=org.apache.log4j.net.SocketAppender
log4j.appender.[myAppender].port=[log4j_port]
log4j.appender.[myAppender].remoteHost=[logstash_host]

There's more info in the logstash docs for their log4j input: http://logstash.net/docs/1.4.2/inputs/log4j