Unable to tell what port Logstash is bound to or listening on when started normally

Question

My logstash version is:

# /opt/logstash/bin/logstash --version
logstash 2.2.4

it is configured to receive input from port 5044 according to the filebeat file:

/etc/logstash/conf.d/02-beats-input.conf

input {
  beats {
    port => 5044
    ssl => false
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

I have set ssl to false as I am not using it

but when I start the logstash service normally with systemctl it start and checking the status confirms it is running

   systemctl status logstash
    ● logstash.service - LSB: Starts Logstash as a daemon.
       Loaded: loaded (/etc/rc.d/init.d/logstash)
       Active: active (exited) since Mon 2016-07-18 19:14:51 BST; 15h ago
         Docs: man:systemd-sysv-generator(8)
      Process: 19965 ExecStop=/etc/rc.d/init.d/logstash stop (code=exited, status=0/SUCCESS)
      Process: 19970 ExecStart=/etc/rc.d/init.d/logstash start (code=exited, status=0/SUCCESS)
...
    logstash started

The problem is that logstash does not seem to be receiving input on port 5044. hosts sending filebeats encounter:

single.go:126: INFO Connecting error publishing events (retrying): dial tcp 192.72.0.92:5044: getsockopt: connection refused

when I check the port

# netstat  -an | grep 5044

I get nothing. So even though logstash is running, I can't tell what port it is bound to and listening on.

Also the firewall is stopped temporarily to investigate this.

The strange thing is that is I run logstash is debug mode like so:

# ./logstash --debug -f /etc/logstash/conf.d/02-beats-input.conf

I can see

# netstat -an | grep 5044
tcp6       0      0 :::5044                 :::*                    LISTEN
tcp6       0      0 192.72.0.92:5044        192.168.36.70:53720     ESTABLISHED
tcp6       0      0 192.72.0.92:5044        192.72.0.90:45980       ESTABLISHED
tcp6       0      0 192.72.0.92:5044        192.72.0.90:45975       ESTABLISHED
tcp6       0      0 192.72.0.92:5044        192.72.0.90:45976       ESTABLISHED

or

# lsof -i :5044
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    15136 root    7u  IPv6 7191510      0t0  TCP *:lxi-evntsvc (LISTEN)
java    15136 root   33u  IPv6 7192379      0t0  TCP hostname:lxi-evntsvc->192.72.0.90:45975 (ESTABLISHED)

and the host sending filebeats can connect

 output.go:87: DBG  output worker: publish 7 events
2016/07/19 10:02:08.017890 client.go:146: DBG  Try to publish 7 events to logstash with window size 10
2016/07/19 10:02:08.038579 client.go:124: DBG  7 events out of 7 events sent to logstash. Continue sending ...
2016/07/19 10:02:08.038615 single.go:135: DBG  send completed

Please help point out what I may be doing wrong with this configuration. Thanks

Answer 1

Based on the hing provided by @LiGhTx117

I think

The startup script used by logstash in:

/etc/init.d/logstash

has the following variables among others:

LS_USER=logstash
LS_GROUP=logstash
LS_HOME=/var/lib/logstash
LS_LOG_DIR=/var/log/logstash
LS_LOG_FILE="${LS_LOG_DIR}/$name.log"
LS_CONF_DIR=/etc/logstash/conf.d

The ownership and permission on these seem to be the issue.

I ensured that the directories where recursively accessible to the user logstash as well as the group logstash

and

Then I also ensured that the log_file: logstash.log was writeable by the user/group logstash

restarted logstash