Why would my REST service .NET clients send every request without authentication headers and then retry it with authentication header?

Question

We happen to run a REST web service with API requiring that clients use Basic authentication. We crafted a set of neat samples in various languages showing how to interface with our service. Now I'm reviewing IIS logs of the service and see that the following pattern happens quite often:

• a request comes, gets rejected with HTTP code 401
• the same request is resent and succeeds

which looks like the first request is sent without Authorization headers and then the second one is sent with the right headers and succeeds. Most of the time the log record contains "user-agent" which is the same string we planted into our .NET sample.

So I assume the problem is with .NET programs only. The problem is not reproduced with our sample code so I assume the users somehow modified the code or wrote their own from scratch.

We tried contacting the users but apparently they don't want to invest time into research. So it'd be nice to find what the most likely scenario is which leads to this behavior of .NET programs.

Why would they do this? Why would they not attach the headers on the first attempt?

Answer 1

This is the default behavior of HttpClient and HttpWebRequest classes which is exposed the following way.

Note: Below text explains suboptimal behavior causing the problem described in the question. Most likely you should not write your code like this. Instead scroll below to the corrected code

In both cases, instantiate a NetworkCredenatial object and set the username and password in there

var credentials = new NetworkCredential( username, password );

If you use HttpWebRequest - set .Credentials property:

webRequest.Credentials = credentials;

If you use HttpClient - pass the credentials object into HttpClientHandler (altered code from here):

var client = new HttpClient(new HttpClientHandler() { Credentials = credentials })

Then run Fiddler and start the request. You will see the following:

• the request is sent without Authorization header
• the service replies with HTTP 401 and WWW-Authenticate: Basic realm="UrRealmHere"
• the request is resent with proper Authorization header (and succeeds)

This behavior is explained here - the client doesn't know in advance that the service requires Basic and tries to negotiate the authentication protocol (and if the service requires Digest sending Basic headers in open is useless and can compromise the client).

Note: Here suboptimal behavior explanation ends and better approach is explained. Most likely you should use code from below instead of code from above.

For cases when it's known that the service requires Basic that extra request can be eliminated the following way:

Don't set .Credentials, instead add the headers manually using code from here. Encode the username and password:

var encoded = Convert.ToBase64String( Encoding.ASCII.GetBytes(
    String.Format( "{0}:{1}", username, password ) ) );

When using HttpWebRequest add it to the headers:

request.Headers.Add( "Authorization", "Basic " + encoded );

and when using HttpClient add it to default headers:

client.DefaultRequestHeaders.Authorization =
    new AuthenticationHeaderValue( "Basic", encoded );

When you do that the request is sent with the right authorization headers every time. Note that you should not set .Credentials, otherwise if the username or password is wrong the same request will be sent twice both time with the wrong credentials and both times of course yielding HTTP 401.