IIS 7 worker process bottleneck, large number of awaiting requests under application pool ASP.NET 3.5 + 2.0

Question

I am using ASP.NET 2.0, .NET 2.0 Framework, and IIS 7. I am seeing a large queue of "requests" appear under the "worker process" option. State recorded appear to be Authenticate Request and Execute Request Handles more than anything else.

I have amended aspnet.config in C:\Windows\Microsoft.NET\Framework64\v2.0.50727 (32 bit path and 64 bit path) to include:

maxConcurrentRequestsPerCPU="50000"
maxConcurrentThreadsPerCPU="0"
requestQueueLimit="50000"

I have amended machine.config in C:\Windows\Microsoft.NET\Framework64\v2.0.50727\CONFIG (32-bit and 64-bit path) to include:

autoConfig="false"
maxIoThreads="100"
maxWorkerThreads="100"
minIoThreads="50"
minWorkerThreads="50"
minFreeThreads="176"
minLocalRequestFreeThreads="152"

Still I get this issue.

The issue manifests itself as a large number of requests in the Worker Process queue.

The number of current connections to the website display 500 when this issue occurs. I don't think I have seen concurrent connections over 500 without this issue occurring.

The web application slows as the requests block.

Refreshing the application pool resolves for a while (as expected) as the load is spread between the two pools.

The application pool in question FIXED REQUEST have been set to refresh on 50000.

Note: .NET 3.5 framework uses 2.0 framework appnet and machine configuration files, I believe.

Server resources (CPU, RAM) are not used to the full potential.

Answer 1

Can I suggest you take a look at the materials on Tess Ferrandez's site If It Is Broken, Fix It You Should.

What you want to be doing is capturing a dump of the worker process using ADPlus at the time you experience a high number of requests in the request queue and when the app starts to grind to a halt.

Once you grabbed this dump you want to load this into WinDBG+SOS and start tracking down the culprit.

Tess has a great series of labs about how to utilise these tools to great effect:

.NET Debugging Demos - Information and setup instructions

If you are able to, then you could also attach a profiler to the application (such as RedGate's Performance Profiler to try and discover the root cause.