Is it really that expensive to increase FileSystemWatcher.InternalBufferSize?
I'm using a FileSystemWatcher to monitor changes in a folder, but as soon as I have more than a few hundred modifications in a short time, I miss some of them because the internal buffer overflows. So I want to increase the InternalBufferSize (I know it won't really solve the problem, but it will make it less likely to occur), but I see this warning in the documentation:
However, increasing buffer size is expensive, because it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small as possible.
So my question is: does it really matter? Most computers today have at least 1GB of RAM, so it seems to me that if I set the buffer size to 1MB (instead of the default 8KB), it shouldn't really matter if 1MB can't be swapped out to disk. Or am I missing something? I don't know much about low level stuff such as paged/non-paged memory, so I'm not sure what the impact would be...
Answer
The memory in which the buffer gets allocated is certainly a precious resource. Windows will not deal with exhausting the memory pool well, drivers will start to fail at random. The size of the pool is dynamically set (but can be changed) and depends on the amount of available RAM.
The default buffer size that FSW asks for is 8192 bytes. Not much on modern machines. The underlying winapi function will not allow you to ask for more than 64KB. An entry is the buffer is 12 bytes plus the length of the file path times two. So worse case is 8192 / (12 + 260*2) = 15 notifications before the buffer runs out. That should work in most circumstances without much trouble, unless you monitor an entire drive or have very high disk traffic in the directory you are watching. In which case asking for a bigger buffer is fair. There is no golden formula, be sure to implement the FileSystemWatcher.Error event so you know that you've got a buffer problem.
In most practical cases you need to deal with the FSW events carefully. They will be raised while a process still has a lock on a file. So doing things like opening or copying the file are troublesome. You deal with that by putting the notifications on a thread-safe queue and use another thread to try to acquire a lock on the file, repeatedly if necessary. Such a queue is now automatically also a very good way to quickly empty the buffer. The only thing you've got to watch for now is that the queue doesn't blow up beyond reasonable proportions that will make your program crash with OOM.