How to force Logstash to reparse a file?

Question

I installed Logstash to parse apache files. It took me quite q while to get the settings right and I always tried on real logs. I noticed (as the documentation says) that logstash "remembers" where it was in a file. Now my setings are Ok and I would like Logstash to "forget". This seems harder than I though. I already did the following:

• used: start_position => "beginning"

• deleted the complete "data" folder from elastissearch (and stopped it first)

• looked at which files where opened by logstash with lsof -p PID and deleted everything which was promising (in my case /tmp/jffi*.tmp)

Still Logstash does not forget and parse only "fresh" files in the folder where the logs are

Any ideas?

Answer 1

By default logstash writes the position is last was on to a logfile which usually resides in $HOME/.sincedb. Logstash can be fooled into believing it never parsed the logfile by specifying /dev/null as sincedb_path.

Here the part of the documentation Input File.

Where to write the since database (keeps track of the current position of monitored log files). Defaults to the value of environment variable "$SINCEDB_PATH" or "$HOME/.sincedb".

Config Example

input {
    file {
        path => "/tmp/logfile_to_analyse"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}