Batch Uploading Huge Sets of Images to Azure Blob Storage

Question

I have about 110,000 images of various formats (jpg, png and gif) and sizes (2-40KB) stored locally on my hard drive. I need to upload them to Azure Blob Storage. While doing this, I need to set some metadata and the blob's ContentType, but otherwise it's a straight up bulk upload.

I'm currently using the following to handle uploading one image at a time (paralleled over 5-10 concurrent Tasks).

static void UploadPhoto(Image pic, string filename, ImageFormat format)
{
    //convert image to bytes
    using(MemoryStream ms = new MemoryStream())
    {
        pic.Save(ms, format);
        ms.Position = 0;

        //create the blob, set metadata and properties
        var blob = container.GetBlobReference(filename);
        blob.Metadata["Filename"] = filename;
        blob.Properties.ContentType = MimeHandler.GetContentType(Path.GetExtension(filename));

        //upload!
        blob.UploadFromStream(ms);
        blob.SetMetadata();
        blob.SetProperties();
    }
}

I was wondering if there was another technique I could employ to handle the uploading, to make it as fast as possible. This particular project involves importing a lot of data from one system to another, and for customer reasons it needs to happen as quickly as possible.

Answer 1

Okay, here's what I did. I tinkered around with running BeginUploadFromStream(), then BeginSetMetadata(), then BeginSetProperties() in an asynchronous chain, paralleled over 5-10 threads (a combination of ElvisLive's and knightpfhor's suggestions). This worked, but anything over 5 threads had terrible performance, taking upwards of 20 seconds for each thread (working on a page of ten images at a time) to complete.

So, to sum up the performance differences:

• Asynchronous: 5 threads, each running an async chain, each working on ten images at a time (paged for statistical reasons): ~15.8 seconds (per thread).
• Synchronous: 1 thread, ten images at a time (paged for statistical reasons): ~3.4 seconds

Okay, that's pretty interesting. One instance uploading blobs synchronously performed 5x better than each thread in the other approach. So, even running the best async balance of 5 threads nets essentially the same performance.

So, I tweaked my image file importing to separate the images into folders containing 10,000 images each. Then I used Process.Start() to launch an instance of my blob uploader for each folder. I have 170,000 images to work with in this batch, so that means 17 instances of the uploader. When running all of those on my laptop, performance across all of them leveled out at ~4.3 seconds per set.

Long story short, instead of trying to get threading working optimally, I just run a blob uploader instance for every 10,000 images, all on the one machine at the same time. Total performance boost?

• Async Attempts: 14-16 hours, based on average execution time when running it for an hour or two.
• Synchronous with 17 separate instances: ~1 hour, 5 minutes.