rsyslog sending badly encoded (corrupted?) data via tcp (receiving using logstash)

encoding logstash rsyslog
Paradise picture Paradise · Jun 30, 2014 · Viewed 7.4k times · Source

My rsyslog logs locally correctly, however I wanted to also receive the logs remotely, so I added the rule:

*.* @@myIP:5141

to the end of my rsyslog.conf

To receive the output, I'm running logstash with the configuration

input { tcp { port => 5141 } }
output { stdout {} }

Logstash expects UTF-8 encoding, however I get the error

Received an event that has a different character encoding than you configured

The messages themselves seem to be garbled, or a mix of encodings, for example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1R\xAB5K\xF6\\\xB9\xB2\xB4\xB1\xAE0\t\u007F\xDF`5\xF6\u0015\xC8)H\xD7H\xCF+&\xD5T5\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

Note some entries are \u00, while others are \x. There are even multiple backslashes.

I was wondering if I messed up the settings somehow, or if there is something between me and the server which is messing up the messages?

I have also tried using the syslog logstash input, which gives the same result

Another example:

\u0016\u0003\u0002\u0000V\u0001\u0000\u0000R\u0003\u0002S\xB1RiZ^\xC3\xD9\u001Cj\a\xD4\xE0\xECr\x8E\xAC\xF5\u001A\xB9+\u07B9\xE5\xF9\xA3''z\u0018}9\u0000\u0000$\u00003\u0000E\u00009\u0000\x88\u0000\u0016\u00002\u0000D\u00008\u0000\x87\u0000\u0013\u0000f\u0000/\u0000A\u00005\u0000\x84\u0000

EDIT: I found the source of my problem, and it was encryption related. Unfortunately I can't disclose what I did to fix it, suffice to say John Petrone's answer below is good start for similar problems that future readers may experience

Answer

docwhat picture docwhat · Nov 14, 2014

So that magic string you're getting back that looks like broken encoding is actually the SSL Handshake request.

I suspect what you've done is (like I just did) misconfigured the tcp input in logstash. Specifically, I forgot to add the ssl_enable => true. So it was listening for normal TCP and got SSL Handshake and dutifully recorded it as garbage.

Related questions

Converting string to byte array in C#

I'm converting something from VB into C#. Having a problem with the syntax of this statement: if ((searchResult.Properties["user"].Count > 0)) { profile.User = System.Text.Encoding.UTF8.GetString(searchResult.Properties["user"][0]); } I then see the following errors: Argument 1: cannot …

Read More
Java URL encoding of query string parameters

Say I have a URL http://example.com/query?q= and I have a query entered by the user such as: random word £500 bank $ I want the result to be a properly encoded URL: http://example.com/query?q=random%20…

Read More
Working with UTF-8 encoding in Python source

Consider: $ cat bla.py u = unicode('d…') s = u.encode('utf-8') print s $ python bla.py File "bla.py", line 1 SyntaxError: Non-ASCII character '\xe2' in file bla.py on line 1, but no encoding declared; see http://…

Read More