Centralized network logging - syslog and alternatives?

logging networking message-queue syslog
Julian Morrison picture Julian Morrison · Nov 27, 2009 · Viewed 11.1k times · Source

At work, we're building a distributed application (possibly across several machines on a LAN, possibly later across several continents on a WAN+VPN). We don't want log files local to each machine (filling up its disk and impossible to view in aggregate) so we need to centralize logging over the network. Most logs won't be important, so UDP is fine for them, but some are money-losing important alerts and must be reliably delivered, implying TCP. We're worried about congesting the network if the logging protocol is too chatty, or dragging the apps to a crawl if it isn't responsive.

Some possibilities I've considered are:

  • syslog (it seems perfect, but my boss has an animus against this so I may not be able to choose it).
  • scribe from facebook (but it seems a bit heavyweight with a server on every machine - not every log message needs ultra-reliability).
  • using a message queue like rabbitmq which can have multiple queues tuned to different levels of transaction safety.
  • worst case, I can write my own from scratch.

Do you have other suggestions? What centralized logging solutions have you used, and how well did they work out?

Edit: I was leaning towards scribe, because its store-and-forward design decouples the running app from network latency. But after struggling to install it, I found that (1) it's not available as a binary package - nowadays that's unforgivable - and (2) it depends intimately on a library (thrift) that isn't available as a binary package either! And worst of all, it wouldn't even compile properly. That is not release quality code, even in open source.

Answer

Ass3mbler picture Ass3mbler · Nov 27, 2009

We have successfully used ZeroMQ for the logs of a distributed application scenario like the yours. It's very reliable and incredibly fast. We moved to ZeroMQ after a not-so-successful implementation with Spread. In our setup a single ZeroMQ server is able to handle more than 70 different logs from a medium to high busy distributed application. It receives data from LAN and via the Internet.

If you need a detailed queue server comparison, look at this page from the Second Life wiki.

Hope it helps!

Related questions

grep a file, but show several surrounding lines?

I would like to grep for a string, but also show the preceding five lines and the following five lines as well as the matched line. How would I be able to do this?

Read More
String formatting: % vs. .format vs. string literal

Python 2.6 introduced the str.format() method with a slightly different syntax from the existing % operator. Which is better and for what situations? Python 3.6 has now introduced another string formatting format of string literals (aka "f" strings) via the syntax f"…

Read More
python exception message capturing

import ftplib import urllib2 import os import logging logger = logging.getLogger('ftpuploader') hdlr = logging.FileHandler('ftplog.log') formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s') hdlr.setFormatter(formatter) logger.addHandler(hdlr) logger.setLevel(logging.INFO) FTPADDR = "some ftp …

Read More