FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory when processing large files with fs

Question

I have a nodeJs script that process a bunch of large .csv files (1.3GB for all). It run for a moment and throw this error:

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

I have tried to put #!/usr/bin/env node --max-old-space-size=4096 at the beginning of my file but this didn't solve my problem...

Tried to google it but nearly no pertinent info about my issue..

Can I flush memory once I do not need file content ? Do I need to allocate more memory to nodeJs ?

Thanks ;)

Here is my code sample:

fs.readdir(dirName, function(err, filenames) {
    if (err) console.error(err);
    else {
        filenames.forEach(function(filename) {
            fs.readFile(dirName + filename, 'utf-8', function(err, content) {
                if (err) console.error(err);
                else processFile(content);
            });
        });
    }
});

Answer 1

I have finally found the solution to the problem ! I need to launch the process adding the --max-old-space-size=8192 param to node process:

node --max-old-space-size=8192 ./myScript.js

I processed my 1.3GB without a problem !!