Max number of concurrent HttpWebRequests

.net httpwebrequest multithreading
Duncan picture Duncan · Sep 1, 2009 · Viewed 61.1k times · Source

I'm stress testing a web app and have set up a windows test program that spins up a number of threads and sends out a web request on each one.

Problem is I get the following output:

01/09/09 11:34:04 Starting new HTTP request on 10
01/09/09 11:34:04 Starting new HTTP request on 11
01/09/09 11:34:04 Starting new HTTP request on 13
01/09/09 11:34:05 Starting new HTTP request on 14
01/09/09 11:34:05 Starting new HTTP request on 11
01/09/09 11:34:05 11 has finished!
01/09/09 11:34:05 Starting new HTTP request on 13
01/09/09 11:34:05 13 has finished!
01/09/09 11:34:05 Starting new HTTP request on 14
01/09/09 11:34:05 14 has finished!
01/09/09 11:34:05 Starting new HTTP request on 11
01/09/09 11:34:05 11 has finished!
01/09/09 11:34:05 Starting new HTTP request on 14
01/09/09 11:34:05 14 has finished!
01/09/09 11:34:05 Starting new HTTP request on 13
01/09/09 11:34:05 13 has finished!
01/09/09 11:34:05 Starting new HTTP request on 15
01/09/09 11:34:06 Starting new HTTP request on 11
01/09/09 11:34:06 11 has finished!
01/09/09 11:34:06 Starting new HTTP request on 14
01/09/09 11:34:06 14 has finished!

which sort of looks like there's a maximum of 5 threads, even if I create 100 as so:

int numberOfThreads = Convert.ToInt32(txtConcurrentThreads.Text);

    List<BackgroundWorker> workers = new List<BackgroundWorker>();

    for (int N = 0; N < numberOfThreads; N++)
    {

        BackgroundWorker worker = new BackgroundWorker();
        worker.DoWork += new DoWorkEventHandler(worker_DoWork);
        worker.RunWorkerCompleted += new RunWorkerCompletedEventHandler(worker_RunWorkerCompleted);
        workers.Add(worker);
    }


    foreach(BackgroundWorker worker in workers)
    {
        worker.RunWorkerAsync();
    }

Can anyone enlighten me as to what is going on?

Thanks

EDIT: If as suggested I sleep for 5 seconds, instead of httpwebrequest, then I do get more threads firing but not as many as I would have expected:

01/09/09 11:56:14 Starting new HTTP request on 7
01/09/09 11:56:14 Starting new HTTP request on 11
01/09/09 11:56:15 Starting new HTTP request on 12
01/09/09 11:56:15 Starting new HTTP request on 13
01/09/09 11:56:16 Starting new HTTP request on 14
01/09/09 11:56:16 Starting new HTTP request on 15
01/09/09 11:56:17 Starting new HTTP request on 16
01/09/09 11:56:17 Starting new HTTP request on 17
01/09/09 11:56:18 Starting new HTTP request on 18
01/09/09 11:56:19 Starting new HTTP request on 7
01/09/09 11:56:19 7 has finished!
01/09/09 11:56:19 Starting new HTTP request on 11
01/09/09 11:56:19 11 has finished!
01/09/09 11:56:19 Starting new HTTP request on 19
01/09/09 11:56:20 Starting new HTTP request on 20
01/09/09 11:56:20 Starting new HTTP request on 12
01/09/09 11:56:20 12 has finished!

It still looks like I'm only getting 2 threads starting every second, which seems mighty slow to me. I suppose the Console.WriteLine could be a problem?

EDIT: I set 

ThreadPool.SetMinThreads(100, 4);

and 

System.Net.ServicePointManager.DefaultConnectionLimit = 100;

and got the following results:

01/09/09 14:00:07 Starting new HTTP request on 11
01/09/09 14:00:07 Starting new HTTP request on 81
01/09/09 14:00:07 Starting new HTTP request on 82
01/09/09 14:00:07 Starting new HTTP request on 79
01/09/09 14:00:07 Starting new HTTP request on 83
01/09/09 14:00:07 Starting new HTTP request on 84
01/09/09 14:00:07 Starting new HTTP request on 85
01/09/09 14:00:07 Starting new HTTP request on 87
01/09/09 14:00:07 Starting new HTTP request on 88
...
01/09/09 14:00:07 84 has finished! Took 323.0323 milliseconds
01/09/09 14:00:08 88 has finished! Took 808.0808 milliseconds
01/09/09 14:00:08 96 has finished! Took 806.0806 milliseconds
01/09/09 14:00:08 94 has finished! Took 806.0806 milliseconds
01/09/09 14:00:08 98 has finished! Took 801.0801 milliseconds
01/09/09 14:00:08 80 has finished! Took 799.0799 milliseconds
01/09/09 14:00:08 86 has finished! Took 799.0799 milliseconds
01/09/09 14:00:08 92 has finished! Took 799.0799 milliseconds
01/09/09 14:00:08 100 has finished! Took 812.0812 milliseconds
01/09/09 14:00:08 82 has finished! Took 1010.101 milliseconds

so was able to push out a whole lot of web requests concurrently. Which seemed to queue (calling out to an STA COM+ server) so that's what I expected.

Thanks for your help

Answer

Jon Skeet picture Jon Skeet · Sep 1, 2009

Are all (or most) of your requests going to the same host by any chance? There's a built-in limit on a per-host basis. You can change this in app.config in the system.Net connectionManagement element.

The other thing is that the thread pool only ramps up its number of threads gradually - it starts a new thread every half second, IIRC. Could that be what you're seeing? Try getting rid of HttpWebRequest from the equation - just sleep for a couple of seconds instead...

I suspect the latter problem is the one you're initially running into, but the first one is going to cause you problems as well.

Related questions

Improving performance of multithreaded HttpWebRequests in .NET

I am trying to measure the throughput of a webservice. In order to do that, I have written a small tool that continuously sends requests and reads responses from a number of threads. The contents of the inner loop of …

Read More
How to make an HTTP POST web request

Canonical How can I make an HTTP request and send some data using the POST method? I can do a GET request, but I have no idea of how to make a POST request.

Read More
Upload files with HTTPWebrequest (multipart/form-data)

Is there any class, library or some piece of code which will help me to upload files with HTTPWebrequest? Edit 2: I do not want to upload to a WebDAV folder or something like that. I want to simulate a browser, …

Read More