azure blob returns 403 forbidden with correct access key

Question

My test site has after a deploy started to get 403 forbidden back when trying to access files from the azure blob storage. This is only a problem on our test environment, the new release works just fine in production. Both production and test is hosted in azure, and both use their own azure blob storage.

I have tried regenerating the access keys for the blob storage, without any luck.
I can use the access keys locally and connect to the test blob storage and access the files just fine.
If i try to change the test environment to use the production blob storage, i still get the 403 forbidden error.

Microsoft.WindowsAzure.Storage.StorageException: The remote server returned an error: (403) Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden. at System.Net.HttpWebRequest.GetResponse() at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 677 --- End of inner exception stack trace --- at Microsoft.WindowsAzure.Storage.Core.Executor.Executor.ExecuteSync[T](RESTCommand1 cmd, IRetryPolicy policy, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Core\Executor\Executor.cs:line 604 at Microsoft.WindowsAzure.Storage.Blob.CloudBlobClient.GetBlobReferenceFromServer(StorageUri blobUri, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlobClient.cs:line 563 at Microsoft.WindowsAzure.Storage.Blob.CloudBlobContainer.GetBlobReferenceFromServer(String blobName, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext) in c:\Program Files (x86)\Jenkins\workspace\release_dotnet_master\Lib\ClassLibraryCommon\Blob\CloudBlobContainer.cs:line 818 at ASP.XYZ in d:\XYZ.cshtml:line 27 Request Information RequestID:fc137321-0001-00ce-02d2-de5736000000 RequestDate:Tue, 06 Jun 2017 14:41:21 GMT StatusMessage:Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

It must be an issue on the azure app service that runs the test site. I just cannot see what it could be.

See gist here for some very simplified code https://gist.github.com/Skaanning/5cddf95a0d1ff210482d99a683e0be9c .

Edit I have multiple other blob stores, but none of them work on the test site. They work fine on other environments though.

I tried scaling it up and down, without any luck. But then i tried changing from 32 bit to 64 bit and now it works! I will keep the question open, in case someone can explain what on earth happened here

Answer 1

var img = container.GetBlobReference("someimage.png"); // this works just fine
var img2 = container.GetBlobReferenceFromServer("someimage.png"); // this throws a 403

The first line will work fine. The reason is that it will not send request to server when we execute GetBlobReference method. When executing GetBlobReferenceFromServer method, it will send a request to server to get the blob content. It will throw 404 not found exception if the blob can't be found in blob server.

My test site has after a deploy started to get 403 forbidden back when trying to access files from the azure blob storage.

The 403 forbidden exception often caused by a wrong access key is used. Please print out the connection string which was used in your test environment.

Trace.Write(CloudConfigurationManager.GetSetting("blob.storage"));

A Azure Storage connection string uses following format. Please make sure you have provided a right account name and key pair for the connection string. If you set account name1 and account key for account name2, it will cause the exception.

DefaultEndpointsProtocol=https;AccountName=[accountName];AccountKey=[accountKey];EndpointSuffix=core.windows.net

Another thing would cause this issue is that the request to your storage server is rejected by your test server. Please check whether you have configured Dynamic IP Security in your web.config.

Edit 2017/6/7 4:26 PM

I suggest you take my upper suggestion. Please print out the connection string at runtime. If you set the connections string in both app setting in Azure portal and web.config. settings in Azure portal will override the settings in web.config.

Please also print out the current date time at runtime to check whether system time of your app instance is right. If the date or time of your system has been changed, 404 Forbidden also will happen.

The storage services ensure that a request is no older than 15 minutes by the time it reaches the service. This guards against certain security attacks, including replay attacks. When this check fails, the server returns response code 403 (Forbidden).

Reference: Authentication for the Azure Storage Services

Edit 2017/6/7 5:55 PM

Have you tried removing all the original files at your test server when deploying your release?