CPU Usage Spikes in WebSphere 6.1

Question

First, just a bit of background:

One of our customers is experiencing CPU usage spikes for WebSphere instances running one of our web apps (other instances with other apps are fine). They have a test environment and a live environment (both iSeries) which both experience the problem - with a single app per instance setup. We have deployed this application locally in our own test environments and also for many other customers all on iSeries with no similar problems.

What's actually happening:

Every one second or so, the CPU usage for the WebSphere process' CPU usage jumps to anywhere from 7%-20% even though there are no requests being processed at the time. Customer has reported seeing spikes as high as 30%. These spikes average out to be 1.5% of CPU overall - the other WebSphere instances typically use 0%-0.1% when idle.

My investigations so far

So, I had a look at the threads. One thread in there test environment was using ~350 CPU cycles per second. A similar thread in their live environment was using ~1500 CPU cycles per second (showing that it has bigger CPU). The call stack for these threads looks like

Type  Program                  Statement         Procedure                    
      QLESPI     QSYS          17                LE_Create_Thread2__FP12crtt >
      QJVALIBJVM QSYS          7                 startThread__FPv             
 J    com/ibm/ws/util/Threa >                    run                          
 J    com/ibm/ws/util/Threa >                    run                          
 J    com/ibm/ws/util/Threa >                    getTask                      
 J    com/ibm/ws/util/Bound >                    poll

The entire class name from the bottom line is com/ibm/ws/util/BoundedBuffer. I asked the customer to do a JVM Dump for me - the only additional information I got from this was the thread name:

Thread:  00002F82 Deferrable Alarm : 11

Now for my questions:

• Can any of you identify the problem, given these symptoms? (Maybe that's a long shot!)
• What is Deferrable Alarm? From the JVM Dump, I can see 4 threads with this name. The other three seem to be doing just fine. By debugging my local WebSphere (on Windows) and adding breakpoints in the BoundedBuffer class, I see that BoudedBuffers are polling and periodically invoking some listener.
• I don't have access to the WebSphere console for the customer machines, and they aren't owning up to having made any config changes. I can ask them to check the console for me though - what should I be asking them to look at?
• I have telnet access to the customer boxes, is there anything else I can investigate here? Looking at the WebSphere profile files, etc? Which files should I be looking at?
• Because the Call Stack and JVM Dump don't explicitly reference our code, is it safe to assume that this is a configuration problem?

It's been a long question, so thanks for reading this far.

30 April Update (1)

This morning I've noticed that this behaviour only happens after the first request of the day has been processed (irrespective of which Web Service is invoked). This points the finger back at our application or Apache Axis. Could it be that this is just normal behaviour?!

30 April Update (2)

So it seems that this CPU activity is some kind of housekeeping activity for the web-container or maybe something within Apache Axis. I've now observed this happening on a few different web-applications on a few different servers. Applications with no web component don't suffer the same additional CPU overhead.

I'd imagine if it is housekeeping work, that "tuning" it somehow could be counter productive - by that, I mean that making the App Server idle better would probably negatively affect the amount of "real" work it can do.

Answer 1

You could try to profile and do heap dumps of the application, that could answer a few questions related to memory and cpu usage.