log4net BufferingForwardingAppender performance issue

Question

EDIT 2 : I have solved the problem (see answer below) Please note that the problem potentially affects all appenders decorated with BufferingForwardingAppender as well as all appenders inheriting from BufferingAppenderSkeleton (respectively : AdoNetAppender, RemotingAppender, SmtpAppender and SmtpPickupDirAppender) *

I was doing some very basic benchs of log4net and I tried to decorate a RollingFileAppender with a BufferingForwardingAppender.

I experience terrible performance going through the BufferingForwardingAppender instead of directly through the RollingFileAppender and I really don't get the reason.

Here is my configuration:

<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
  <file value="c:\" />
  <appendToFile value="false" />
  <rollingStyle value="Composite" />
  <datePattern value="'.'MMdd-HH'.log'" />
  <maxSizeRollBackups value="168" />
  <staticLogFileName value="false" />
  <layout type="log4net.Layout.PatternLayout">      
    <conversionPattern value="%date [%thread] %-5level %logger - %message%newline" />
  </layout>
</appender>

<appender name="BufferingForwardingAppender" type="log4net.Appender.BufferingForwardingAppender">
  <bufferSize value="512" />
  <appender-ref ref="RollingLogFileAppender" />
</appender>

<root>
  <level value="DEBUG" />
  <appender-ref ref="BufferingForwardingAppender" />    
</root>

And here is the benchmark (very simple code):

var stopWatch = new Stopwatch();
stopWatch.Start();            
for (int i = 0; i < 100000; i++)            
{
   Log.Debug("Hello");
}
stopWatch.Stop();
Console.WriteLine("Done in {0} ms", stopWatch.ElapsedMilliseconds);

Going directly through RollingFileAppender the output is:

Done in 511 ms

Whereas going through the BufferingForwardingAppender decorating the RollingFileAppender :

Done in 14261 ms

That's approx 30 times slower.

I thought I would gain some speed by buffering a certain amount of log before writing them to the file, however for some reason it gets things much worse.

Seems to me like the configuration is OK, so this is really weird.

Anyone got a clue?

Thanks!

EDIT 1 :

The behavior is strictly the same by wrapping/decorating a FileAppender or even ConsoleAppender (still there is an example of basic BufferingForwardingAppender wrapping/decorating ConsoleAppender in log4net official config samples .. and nothing specific mentioned dealing with performance).

After some investigation/profiling, I can see that the majority of the time is spoiled inside the BufferingForwardingAppender more specifically in a call to WindowsIdentity.GetCurrent() ... being called EACH time we make a call to Log.Debug() .. in the previous sample (100K times in the sample source above).

Calls to this method are known to be very costly and should be avoided or minimized, I really don't get why it gets called for each log event. Am I really completely misconfiguring something / not seeing something evident, or is that a bug somehow somewhere, this is what I am trying to figure out now...

The partial call stack is :

• AppenderSkeleton.DoAppend
• BufferingAppenderSkeleton.Append
• LoggingEvent.FixVolatileData
• LoggingEvent.get_UserName()

A call to get_LocationInformation() is also done in FixVolatileData, incurring an high perf cost as well (capture the stack trace each time).

I am now trying to understand why this extremely costly FixVolatileData call (at least for the fix asked) happens for each log event in this context whereas going directly through the wrapped appender (directly through ConsoleAppender/FileAppender ..) does not perform this kind of operation.

Upcoming update to follow, unless someone got an answer to all of this ;)

Thanks!

Answer 1

I found out the issue.

The BufferingForwardingAppender is inheriting from BufferingAppenderSkeleton (as are other appenders making use of logging events buffering such as AdoNetAppender, RemotingAppender, SmtpAppender ..).

The BufferingAppenderSkeleton is actually buffering logging events before actually delivering them to the target appender once a certain condition is met (buffer full for example).

According to documentation of the LoggingEvent class (representing a logging event, and containing all values (message, threadid ...) of the event) :

Some logging events properties are considered "volatile", that is the values are correct at the time the event is delivered to appenders, but will not be consistent at any time afterwards. If an event is to be stored and the processed at a later time, these volatile values must be fixed bycalling FixVolatileData. There is a performance penalty incurred by calling FixVolatileData but is is essential to maintain data consistency

These "volatile" properties are represented by the FixFlags enumeration containing flags such as Message, ThreadName, UserName, Identity ... so all volatile properties. It also contains the flag "None" (fix no properties), "All" (fix all properties) and "Partial" (fix only a certain predefine dset of properties).

Whem the BufferingAppenderSkeleton is instanciated, by DEFAULT it sets the fixing to "All" meaning that all "volatile" properties should be fixed.

In that context, for each LoggingEvent appended into the BufferingAppenderSkeleton, ALL "volatile" properties will be fixed before the event is inserted in the buffer. This includes the properties Identity (username) and LocationInformation (stack trace) even if these properties are not included in the layout (but I guess it makes some kind of sense if the layout is changed to include these properties at a later time while a buffer has been already been filled with LoggingEvents).

However in my case this really HURTS performance. I am not including the Identity and LocationInformation in my layout and don't plan to (mainly for performance issues)

Now for the solution ...

There are two properties in BufferingAppenderSkeleton which can be used to control the FixFlags flag value of the BufferingAppenderSkeleton (once again by default it is set to "ALL" which is not very nice !). These two properties are Fix (FixFlags type) and OnlyFixPartialEventData (bool type).

For a fine tune of the flag value or to disable all fix, the Fix property should be used. For a specific partial predefined combination of flags (not including Identity or LocationInfo), the OnlyFixPartialEventData can be used instead by setting it to "true".

If I reuse the configuration sample above (in my question), the only change made to the configuration to unleash performance is indicated below:

<appender name="BufferingForwardingAppender" type="log4net.Appender.BufferingForwardingAppender">
  <bufferSize value="512" />
  <appender-ref ref="RollingLogFileAppender" />
  <Fix value="0"/> <!-- Set Fix flag to NONE -->
</appender>

Using this modified configuration, the benchmark code execution presented in my question above, is dropping from approx 14000ms to 230ms (60X faster) ! And if I use <OnlyFixPartialEventData value="true"/> instead of disabling all fix it is taking approx 350ms.

Sadly, this flag is not very well documented (except in the SDK documentation, a little bit) .. so I had to dig deep into log4net sources to find the issue.

This is particularly problematic especially in the "reference" configuration samples, this flag appears nowhere (http://logging.apache.org/log4net/release/config-examples.html). So the samples provided for BufferingForwardingAppender, and AdoNetAppender (and other appenders inheriting from BufferingAppenderSkeleton) will give TERRIBLE performance to users, even if the layout they are using is pretty minimal.