Is async HttpClient from .Net 4.5 a bad choice for intensive load applications?

Question

I recently created a simple application for testing the HTTP call throughput that can be generated in an asynchronous manner vs a classical multithreaded approach.

The application is a able to perform a predefined number of HTTP calls and at the end it displays the total time needed to perform them. During my tests, all HTTP calls were made to my local IIS sever and they retrieved a small text file (12 bytes in size).

The most important part of the code for the asynchronous implementation is listed below:

public async void TestAsync()
{
    this.TestInit();
    HttpClient httpClient = new HttpClient();

    for (int i = 0; i < NUMBER_OF_REQUESTS; i++)
    {
        ProcessUrlAsync(httpClient);
    }
}

private async void ProcessUrlAsync(HttpClient httpClient)
{
    HttpResponseMessage httpResponse = null;

    try
    {
        Task<HttpResponseMessage> getTask = httpClient.GetAsync(URL);
        httpResponse = await getTask;

        Interlocked.Increment(ref _successfulCalls);
    }
    catch (Exception ex)
    {
        Interlocked.Increment(ref _failedCalls);
    }
    finally
    { 
        if(httpResponse != null) httpResponse.Dispose();
    }

    lock (_syncLock)
    {
        _itemsLeft--;
        if (_itemsLeft == 0)
        {
            _utcEndTime = DateTime.UtcNow;
            this.DisplayTestResults();
        }
    }
}

The most important part of the multithreading implementation is listed below:

public void TestParallel2()
{
    this.TestInit();
    ServicePointManager.DefaultConnectionLimit = 100;

    for (int i = 0; i < NUMBER_OF_REQUESTS; i++)
    {
        Task.Run(() =>
        {
            try
            {
                this.PerformWebRequestGet();
                Interlocked.Increment(ref _successfulCalls);
            }
            catch (Exception ex)
            {
                Interlocked.Increment(ref _failedCalls);
            }

            lock (_syncLock)
            {
                _itemsLeft--;
                if (_itemsLeft == 0)
                {
                    _utcEndTime = DateTime.UtcNow;
                    this.DisplayTestResults();
                }
            }
        });
    }
}

private void PerformWebRequestGet()
{ 
    HttpWebRequest request = null;
    HttpWebResponse response = null;

    try
    {
        request = (HttpWebRequest)WebRequest.Create(URL);
        request.Method = "GET";
        request.KeepAlive = true;
        response = (HttpWebResponse)request.GetResponse();
    }
    finally
    {
        if (response != null) response.Close();
    }
}

Running the tests revealed that the multithreaded version was faster. It took it around 0.6 seconds to complete for 10k requests, while the async one took around 2 seconds to complete for the same amount of load. This was a bit of a surprise, because I was expecting the async one to be faster. Maybe it was because of the fact that my HTTP calls were very fast. In a real world scenario, where the server should perform a more meaningful operation and where there should also be some network latency, the results might be reversed.

However, what really concerns me is the way HttpClient behaves when the load is increased. Since it takes it around 2 seconds to deliver 10k messages, I thought it would take it around 20 seconds to deliver 10 times the number of messages, but running the test showed that it needs around 50 seconds to deliver the 100k messages. Furthermore, it usually takes it more than 2 minutes to deliver 200k messages and often, a few thousands of them (3-4k) fail with the following exception:

An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full.

I checked the IIS logs and operations that failed never got to the server. They failed within the client. I ran the tests on a Windows 7 machine with the default range of ephemeral ports of 49152 to 65535. Running netstat showed that around 5-6k ports were being used during tests, so in theory there should have been many more available. If the lack of ports was indeed the cause of the exceptions it means that either netstat didn't properly report the situation or HttClient only uses a maximum number of ports after which it starts throwing exceptions.

By contrast, the multithread approach of generating HTTP calls behaved very predictable. I took it around 0.6 seconds for 10k messages, around 5.5 seconds for 100k messages and as expected around 55 seconds for 1 million messages. None of the messages failed. Further more, while it ran, it never used more than 55 MB of RAM (according to Windows Task Manager). The memory used when sending messages asynchronously grew proportionally with the load. It used around 500 MB of RAM during the 200k messages tests.

I think there are two main reasons for the above results. The first one is that HttpClient seems to be very greedy in creating new connections with the server. The high number of used ports reported by netstat means that it probably doesn't benefit much from HTTP keep-alive.

The second is that HttpClient doesn't seem to have a throttling mechanism. In fact this seems to be a general problem related to async operations. If you need to perform a very large number of operations they will all be started at once and then their continuations will be executed as they are available. In theory this should be ok, because in async operations the load is on external systems but as proved above this is not entirely the case. Having a big number of requests started at once will increase the memory usage and slow down the entire execution.

I managed to obtain better results, memory and execution time wise, by limiting the maximum number of asynchronous requests with a simple but primitive delay mechanism:

public async void TestAsyncWithDelay()
{
    this.TestInit();
    HttpClient httpClient = new HttpClient();

    for (int i = 0; i < NUMBER_OF_REQUESTS; i++)
    {
        if (_activeRequestsCount >= MAX_CONCURENT_REQUESTS)
            await Task.Delay(DELAY_TIME);

        ProcessUrlAsyncWithReqCount(httpClient);
    }
}

It would be really useful if HttpClient included a mechanism for limiting the number of concurrent requests. When using the Task class (which is based on the .Net thread pool) throttling is automatically achieved by limiting the number of concurrent threads.

For a complete overview, I have also created a version of the async test based on HttpWebRequest rather than HttpClient and managed to obtain much better results. For a start, it allows setting a limit on the number of concurrent connections (with ServicePointManager.DefaultConnectionLimit or via config), which means that it never ran out of ports and never failed on any request (HttpClient, by default, is based on HttpWebRequest, but it seems to ignore the connection limit setting).

The async HttpWebRequest approach was still about 50 - 60% slower than the multithreading one, but it was predictable and reliable. The only downside to it was that it used a huge amount of memory under big load. For example it needed around 1.6 GB for sending 1 million requests. By limiting the number of concurrent requests (like I did above for HttpClient) I managed to reduce the used memory to just 20 MB and obtain an execution time just 10% slower than the multithreading approach.

After this lengthy presentation, my questions are: Is the HttpClient class from .Net 4.5 a bad choice for intensive load applications? Is there any way to throttle it, which should fix the problems I mention about? How about the async flavor of HttpWebRequest?

Update (thanks @Stephen Cleary)

As it turns out, HttpClient, just like HttpWebRequest (on which it is based by default), can have its number of concurrent connections on the same host limited with ServicePointManager.DefaultConnectionLimit. The strange thing is that according to MSDN, the default value for the connection limit is 2. I also checked that on my side using the debugger which pointed that indeed 2 is the default value. However, it seems that unless explicitly setting a value to ServicePointManager.DefaultConnectionLimit, the default value will be ignored. Since I didn't explicitly set a value for it during my HttpClient tests I thought it was ignored.

After setting ServicePointManager.DefaultConnectionLimit to 100 HttpClient became reliable and predictable (netstat confirms that only 100 ports are used). It is still slower than async HttpWebRequest (by about 40%), but strangely, it uses less memory. For the test which involves 1 million requests, it used a maximum of 550 MB, compared to 1.6 GB in the async HttpWebRequest.

So, while HttpClient in combination ServicePointManager.DefaultConnectionLimit seem to ensure reliability (at least for the scenario where all the calls are being made towards the same host), it still looks like its performance is negatively impacted by the lack of a proper throttling mechanism. Something that would limit the concurrent number of requests to a configurable value and put the rest in a queue would make it much more suitable for high scalability scenarios.

Answer 1

Besides the tests mentioned in the question, I recently created some new ones involving much fewer HTTP calls (5000 compared to 1 million previously) but on requests that took much longer to execute (500 milliseconds compared to around 1 millisecond previously). Both tester applications, the synchronously multithreaded one (based on HttpWebRequest) and asynchronous I/O one (based on HTTP client) produced similar results: about 10 seconds to execute using around 3% of the CPU and 30 MB of memory. The only difference between the two testers was that the multithreaded one used 310 threads to execute, while the asynchronous one just 22. So in an application that would have combined both I/O bound and CPU bound operations the asynchronous version would have produced better results because there would have been more CPU time available for the threads performing CPU operations, which are the ones that actually need it (threads waiting for I/O operations to complete are just wasting).

As a conclusion to my tests, asynchronous HTTP calls are not the best option when dealing with very fast requests. The reason behind that is that when running a task that contains an asynchronous I/O call, the thread on which the task is started is quit as soon the as the asynchronous call is made and the rest of the task is registered as a callback. Then, when the I/O operation completes, the callback is queued for execution on the first available thread. All this creates an overhead, which makes fast I/O operations to be more efficient when executed on the thread that started them.

Asynchronous HTTP calls are a good option when dealing with long or potentially long I/O operations because it doesn't keep any threads busy on waiting for the I/O operations to complete. This decreases the overall number of threads used by an application allowing more CPU time to be spent by CPU bound operations. Furthermore, on applications that only allocate a limited number of threads (like it is the case with web applications), asynchronous I/O prevents thread pool thread depletion, which can happen if performing I/O calls synchronously.

So, async HttpClient is not a bottleneck for intensive load applications. It is just that by its nature it is not very well suited for very fast HTTP requests, instead it is ideal for long or potentially long ones, especially inside applications that only have a limited number of threads available. Also, it is a good practice to limit concurrency via ServicePointManager.DefaultConnectionLimit with a value that high enough to ensure a good level of parallelism, but low enough to prevent ephemeral port depletion. You can find more details on the tests and conclusions presented for this question here.