File input not working for logstash

Question

I get the correct output when I use stdin as an input stream. But whenever I use the file as an input, the output freezes after the following message.

"Using milestone 2 input plugin 'file'. This plugin should be stable but if you see strange behavior, please let us know."

Here is my config file.

input {
  file {
    path => ["c:/users/a/b/c/logstash-1.4.1/bin/logs/logfile.log"]
    start_position => beginning
  }
}

filter {
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{MY_DATE:my_date}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{DATE:date}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{TIME:time}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{LOG_LEVEL:log_level}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{SERVER:server}"]
        }
    grok {
                patterns_dir => "./patterns"
                break_on_match => "false"
                match => ["message", "%{CLASS_NAME:class_name}"]
        }

}

output {
  stdout { codec => rubydebug }
  elasticsearch { host => localhost }
 }

Is my file path in the wrong format?

Answer 1

Do you write new logs to your log file?

The start_position option only modifies “first contact” situations where a file is new and not seen before. It is because Logstash will save a sincedb for each file to keep track the current position of monitored log files. So, next time when you restart Logstash, Logstash will start monitor the file based on the sincedb record and the start_position will not work.

So, if you want to import old logs, you have to delete all the .sincedb file before you start logstash and add the start_position option.