StringBuilder Class OutOfMemoryException

Question

I have written following function

public void TestSB()
{
  string str = "The quick brown fox jumps over the lazy dog.";
  StringBuilder sb = new StringBuilder();
  int j = 0;
  int len = 0;

  try
  {
     for (int i = 0; i < (10000000 * 2); i++)
     {
        j = i;
        len = sb.Length;
        sb.Append(str);
     }

    Console.WriteLine("Success ::" + sb.Length.ToString());
  }
  catch (Exception ex)
  {
      Console.WriteLine(
          ex.Message + " :: " + j.ToString() + " :: " + len.ToString());
  }
}

Now I suppose, that StringBuilder has the capacity to take over 2 billion character (2,147,483,647 to be precise).

But when I ran the above function it gave System.OutOfMemoryException just on reaching the capacity of about 800 million. Moreover, I am seeing widely different result on different PC having same memory and similar amount of load.

Can anyone please provide or explain me the reason for this?

Answer 1

Each character requires 2 bytes (as a char in .NET is a UTF-16 code unit). So by the time you've reached 800 million characters, that's 1.6GB of contiguous memory required¹. Now when the StringBuilder needs to resize itself, it has to create another array of the new size (which I believe tries to double the capacity) - which means trying to allocate a 3.2GB array.

I believe that the CLR (even on 64-bit systems) can't allocate a single object of more than 2GB in size. (That certainly used to be the case.) My guess is that your StringBuilder is trying to double in size, and blowing that limit. You may be able to get a little higher by constructing the StringBuilder with a specific capacity - a capacity of around a billion may be feasible.

In the normal course of things this isn't a problem, of course - even strings requiring hundreds of megs are rare.

---

¹ I believe the implementation of StringBuilder actually changed in .NET 4 to use fragments in some situations - but I don't know the details. So it may not always need contiguous memory while still in builder form... but it would if you ever called ToString.