Upon restarting Logstash, at times observed that Logstash duplicates the log events. Was wondering as to what would be the right way to apply start_position
, sincedb_path
, sincedb_write_interval
configuration options.
/home/tom/testData/*.log
XXX.log
file is renamed to XXX-<date>.log
and a new XXX.log
file is created. In this case name doesn't change, but the inode changes.Would highly appreciate if anyone can throw some light on this.
input {
file {
path => "/home/tom/testData/*.log"
type => "log"
start_position => "beginning"
sincedb_path => "/persistent/loc"
sincedb_write_interval => 10
}
}
start_position (beginning or end) is only used for files that have not yet been seen by logstash. The only reason to use 'beginning' is when you're trying to load older files.
sincedb_path just needs to be a directory where logstash has write permission for the registry.
sincedb_write_interval defines how often logstash should write the sincedb registry. A larger value puts you at risk in logstash were to crash.
When you have multiple files that match your glob, logstash tracks them separately by having multiple entries in the registry.
The registry contains the inode number, so logstash knows what to do in that type of rotation.