R multicore mcfork(): Unable to fork: Cannot allocate memory

Question

I'm getting the titular error:

mcfork(): Unable to fork: Cannot allocate memory

after trying to run a function with mcapply, but top says I'm at 51%

This is on an EC2 instance, but I do have up-to-date R.

Does anyone know what else can cause this error?

Thanks,

-N

Answer 1

The issue might be exactly what the error message suggests: there isn't enough memory to fork and create parallel processes.

R essentially needs to create a copy of everything that's in memory for each individual process (to my knowledge it doesn't utilize shared memory). If you are already using 51% of your RAM with a single process, then you don't have enough memory to create a second process since that would required 102% of your RAM in total.

Try:

1. Using fewer cores - If you were trying to use 4 cores, it's possible you have enough RAM to support 3 parallel threads, but not 4. registerDoMC(2), for example, will set the number of parallel threads to 2 (if you are using the doMC parallel backend).
2. Using less memory - without seeing the rest of your code, it's hard to suggest ways to accomplish this. One thing that might help is figuring out which R objects are taking up all the memory (Determining memory usage of objects?) and then removing any objects from memory that you don't need (rm(my_big_object))
3. Adding more RAM - if all else fails, throw hardware at it so you have more capacity.
4. Sticking to single threading - multithreaded processing in R is a tradeoff of CPU and memory. It sounds like in this case you may not have enough memory to support the CPU power you have, so the best course of action might be to just stick to a single core.