Using nxlog to ship logs in to logstash from Windows using om_ssl

Question

I have been looking at options to ship logs from Windows, I have already got logstash set up, and I currently ship logs from Linux (CentOS) servers to my ELK stack using the logstash-forwarder and ssl encryption.

For compliance reasons encryption is pretty much essential in this environment.

I was hoping to use logstash-forwarder in Windows as well, but after compiling with Go I ran in to issues shipping Event Logs, and I found some people saying that it wasn't possible because of file locking issues, which the logstash-forwarder people appear to be working on, but I can't really wait.

Anyway, eventually I found out that nxlog seems to be able to ship logs in an encrypted format using ssl, I've found a few posts about similar topics and while I've learned quite a bit about how to ship the logs across and how to set up nxlog, I am still at a loss with how to set up logstash to accept the logs so I can process them.

I've asked in the #nxlog and #logstash irc channels, and got some confirmation in #nxlog that it is possible, no further information on how it should be configured.

Anyway, I have taken the crt file created for use with my logstash-forwarder (I will create a new one if needed when I am happy that this will work) and renamed it with a pem extension, which I believe should work as it is readable in ASCII format. I have created the environment variable for %CERTDIR% and put my file in there, I have written the following config file for nxlog from the other articles I have read, I think it is right, but I am not 100% sure:

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/nxlog-docs/en/nxlog-reference-manual.html

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

# Enable json extension
<Extension json>
    Module xm_json
</Extension>

# Nxlog internal logs
<Input internal>
    Module im_internal
    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

# Windows Event Log
<Input eventlog>
  # Uncomment im_msvistalog for Windows Vista/2008 and later
    Module im_msvistalog
  # Uncomment im_mseventlog for Windows XP/2000/2003
  # Module im_mseventlog
    Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

<Output sslout>
    Module          om_ssl
    Host            lumberjack.domain.com
    Port            5000
    CertFile        %CERTDIR%/logstash-forwarder.crt
    AllowUntrusted  TRUE
    OutputType      Binary
</Output>

<Route 1>
    Path     eventlog, internal => sslout
</Route>

What I want to know is what input format to use in logstash I have tried shipping logs in to a lumberjack input type (using the same config as my logstash-forwarders use) with the following config:

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

But when the service started I get the following in the nxlog logfiles:

2014-11-06 21:16:20 INFO connecting to lumberjack.domain.com:5000
2014-11-06 21:16:20 INFO nxlog-ce-2.8.1248 started
2014-11-06 21:16:21 INFO successfully connected to lumberjack.domain.com:5000
2014-11-06 21:16:22 INFO remote closed SSL socket
2014-11-06 21:16:22 INFO reconnecting in 1 seconds
2014-11-06 21:16:23 INFO connecting to lumberjack.domain.com:5000
2014-11-06 21:16:24 INFO reconnecting in 2 seconds
2014-11-06 21:16:24 ERROR couldn't connect to ssl socket on lumberjack.antmarketing.com:5000; No connection could be made because the target machine actively refused it.

When I turned the logging up to DEBUG I see a massive amount of logs flying through, but I think the key part is:

2014-11-06 21:20:18 ERROR Exception was caused by "rv" at om_ssl.c:532/io_err_handler(); [om_ssl.c:532/io_err_handler()] -; [om_ssl.c:501/om_ssl_connect()] couldn't connect to ssl socket on lumberjack.domain.com:5000; No connection could be made because the target machine actively refused it.

I assume this points to me using the wrong input method on logstash, but I guess it could also be an issue with my ssl certs or the way it is configured. I don't appear to be getting any logs on the logstash server being generated at the time I make the connection from my Windows machine.

Answer 1

Thanks to b0ti for the help, there were a number of issues, my logstash config was crashing the service, but I also had issues with my nxlog setup as well as my ssl certs being set up in the correct way.

I found this post about creating ssl certs, which covers the way they are set up really nicely for self signed certs for use as a web service.

The main thing wrong with nxlog was as b0ti pointed out I was trying to ship in binary when that will only work when shipping to nxlog server. I also noticed in the docs that the default for AllowUntrusted is false, so I just had to delete it once I was happy ssl was working.

<Output sslout>
    Module          om_ssl
    Host            lumberjack.domain.com
    Port            5001
    CAFile          %CERTDIR%\nxlog-ca.crt
    OutputType      LineBased
</Output>

Creating the CA key, and secure it as this needs to be kept secret (cd to /etc/pki/tls):

certtool --generate-privkey --bits 2048 --outfile private/nxlog-ca.key
chown logstash:logstash private/nxlog-ca.key
chmod 600 private/nxlog-ca.key

And then Self Signed CA Cert, which will need to be transferred to your clients:

certtool --generate-self-signed --load-privkey private/nxlog-ca.key --bits 2048 --template nxlog-ca-rules.cnf --outfile certs/nxlog-ca.crt

The cnf file is standard only with this option modified:

# Whether this is a CA certificate or not
ca

The logstash input method:

input {
  tcp {
    port => 5001
    type => "nxlogs"
    ssl_cacert => "/etc/pki/tls/certs/nxlog-ca.crt"
    ssl_cert => "/etc/pki/tls/certs/nxlog.crt"
    ssl_key => "/etc/pki/tls/private/nxlog.key"
    ssl_enable => true
    format => 'json'
  }
}

Generate the private key:

certtool --generate-privkey --bits 2048 --outfile private/nxlog.key
chown logstash:logstash private private/nxlog.key
chmod 600 private/nxlog.key

Generate the CSR (Certificate Signing Request):

certtool --generate-request --bits 2048 --load-privkey private/nxlog.key --outfile private/nxlog.csr

Sign the Cert with the CA private key

certtool --generate-certificate --bits 2048 --load-request private/nxlog.csr --outfile certs/nxlog.crt --load-ca-certificate certs/nxlog-ca.crt --load-ca-privkey private/nxlog-ca.key --template nxlog-rules.cnf

Again the only important part over the standard inputs for the cnf file will be:

# Whether this certificate will be used to encrypt data (needed
# in TLS RSA ciphersuites). Note that it is preferred to use different
# keys for encryption and signing.
encryption_key

# Whether this certificate will be used for a TLS client
tls_www_client

I've tested this and it works well, I just need to get the filters set up now