MemoryCache does not obey memory limits in configuration

Question

I’m working with the .NET 4.0 MemoryCache class in an application and trying to limit the maximum cache size, but in my tests it does not appear that the cache is actually obeying the limits.

I'm using the settings which, according to MSDN, are supposed to limit the cache size:

1. CacheMemoryLimitMegabytes: The maximum memory size, in megabytes, that an instance of an object can grow to."
2. PhysicalMemoryLimitPercentage: "The percentage of physical memory that the cache can use, expressed as an integer value from 1 to 100. The default is zero, which indicates that MemoryCache instances manage their own memory¹ based on the amount of memory that is installed on the computer." ^{1. This is not entirely correct-- any value below 4 is ignored and replaced with 4.}

I understand that these values are approximate and not hard limits as the thread that purges the cache is fired every x seconds and is also dependent on the polling interval and other undocumented variables. However even taking into account these variances, I'm seeing wildly inconsistent cache sizes when the first item is being evicted from the cache after setting CacheMemoryLimitMegabytes and PhysicalMemoryLimitPercentage together or singularly in a test app. To be sure I ran each test 10 times and calculated the average figure.

These are the results of testing the example code below on a 32-bit Windows 7 PC with 3GB of RAM. Size of the cache is taken after the first call to CacheItemRemoved() on each test. (I am aware the actual size of cache will be larger than this)

MemLimitMB    MemLimitPct     AVG Cache MB on first expiry    
   1            NA              84
   2            NA              84
   3            NA              84
   6            NA              84
  NA             1              84
  NA             4              84
  NA            10              84
  10            20              81
  10            30              81
  10            39              82
  10            40              79
  10            49              146
  10            50              152
  10            60              212
  10            70              332
  10            80              429
  10           100              535
 100            39              81
 500            39              79
 900            39              83
1900            39              84
 900            41              81
 900            46              84

 900            49              1.8 GB approx. in task manager no mem errros
 200            49              156
 100            49              153
2000            60              214
   5            60              78
   6            60              76
   7           100              82
  10           100              541

Here is the test application:

using System;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Linq;
using System.Runtime.Caching;
using System.Text;
namespace FinalCacheTest
{       
    internal class Cache
    {
        private Object Statlock = new object();
        private int ItemCount;
        private long size;
        private MemoryCache MemCache;
        private CacheItemPolicy CIPOL = new CacheItemPolicy();

        public Cache(long CacheSize)
        {
            CIPOL.RemovedCallback = new CacheEntryRemovedCallback(CacheItemRemoved);
            NameValueCollection CacheSettings = new NameValueCollection(3);
            CacheSettings.Add("CacheMemoryLimitMegabytes", Convert.ToString(CacheSize)); 
            CacheSettings.Add("physicalMemoryLimitPercentage", Convert.ToString(49));  //set % here
            CacheSettings.Add("pollingInterval", Convert.ToString("00:00:10"));
            MemCache = new MemoryCache("TestCache", CacheSettings);
        }

        public void AddItem(string Name, string Value)
        {
            CacheItem CI = new CacheItem(Name, Value);
            MemCache.Add(CI, CIPOL);

            lock (Statlock)
            {
                ItemCount++;
                size = size + (Name.Length + Value.Length * 2);
            }

        }

        public void CacheItemRemoved(CacheEntryRemovedArguments Args)
        {
            Console.WriteLine("Cache contains {0} items. Size is {1} bytes", ItemCount, size);

            lock (Statlock)
            {
                ItemCount--;
                size = size - 108;
            }

            Console.ReadKey();
        }
    }
}

namespace FinalCacheTest
{
    internal class Program
    {
        private static void Main(string[] args)
        {
            int MaxAdds = 5000000;
            Cache MyCache = new Cache(1); // set CacheMemoryLimitMegabytes

            for (int i = 0; i < MaxAdds; i++)
            {
                MyCache.AddItem(Guid.NewGuid().ToString(), Guid.NewGuid().ToString());
            }

            Console.WriteLine("Finished Adding Items to Cache");
        }
    }
}

Why is MemoryCache not obeying the configured memory limits?

Answer 1

Wow, so I just spent entirely too much time digging around in the CLR with reflector, but I think I finally have a good handle on what's going on here.

The settings are being read in correctly, but there seems to be a deep-seated problem in the CLR itself that looks like it will render the memory limit setting essentially useless.

The following code is reflected out of the System.Runtime.Caching DLL, for the CacheMemoryMonitor class (there is a similar class that monitors physical memory and deals with the other setting, but this is the more important one):

protected override int GetCurrentPressure()
{
  int num = GC.CollectionCount(2);
  SRef ref2 = this._sizedRef;
  if ((num != this._gen2Count) && (ref2 != null))
  {
    this._gen2Count = num;
    this._idx ^= 1;
    this._cacheSizeSampleTimes[this._idx] = DateTime.UtcNow;
    this._cacheSizeSamples[this._idx] = ref2.ApproximateSize;
    IMemoryCacheManager manager = s_memoryCacheManager;
    if (manager != null)
    {
      manager.UpdateCacheSize(this._cacheSizeSamples[this._idx], this._memoryCache);
    }
  }
  if (this._memoryLimit <= 0L)
  {
    return 0;
  }
  long num2 = this._cacheSizeSamples[this._idx];
  if (num2 > this._memoryLimit)
  {
    num2 = this._memoryLimit;
  }
  return (int) ((num2 * 100L) / this._memoryLimit);
}

The first thing you might notice is that it doesn't even try to look at the size of the cache until after a Gen2 garbage collection, instead just falling back on the existing stored size value in cacheSizeSamples. So you won't ever be able to hit the target right on, but if the rest worked we would at least get a size measurement before we got in real trouble.

So assuming a Gen2 GC has occurred, we run into problem 2, which is that ref2.ApproximateSize does a horrible job of actually approximating the size of the cache. Slogging through CLR junk I found that this is a System.SizedReference, and this is what it's doing to get the value (IntPtr is a handle to the MemoryCache object itself):

[SecurityCritical]
[MethodImpl(MethodImplOptions.InternalCall)]
private static extern long GetApproximateSizeOfSizedRef(IntPtr h);

I'm assuming that extern declaration means that it goes diving into unmanaged windows land at this point, and I have no idea how to start finding out what it does there. From what I've observed though it does a horrible job of trying to approximate the size of the overall thing.

The third noticeable thing there is the call to manager.UpdateCacheSize which sounds like it should do something. Unfortunately in any normal sample of how this should work s_memoryCacheManager will always be null. The field is set from the public static member ObjectCache.Host. This is exposed for the user to mess with if he so chooses, and I was actually able to make this thing sort of work like it's supposed to by slopping together my own IMemoryCacheManager implementation, setting it to ObjectCache.Host, and then running the sample. At that point though, it seems like you might as well just make your own cache implementation and not even bother with all this stuff, especially since I have no idea if setting your own class to ObjectCache.Host (static, so it affects every one of these that might be out there in process) to measure the cache could mess up other things.

I have to believe that at least part of this (if not a couple parts) is just a straight up bug. It'd be nice to hear from someone at MS what the deal was with this thing.

TLDR version of this giant answer: Assume that CacheMemoryLimitMegabytes is completely busted at this point in time. You can set it to 10 MB, and then proceed to fill up the cache to ~2GB and blow an out of memory exception with no tripping of item removal.