Buffer entire file in memory with Node.js

node.js stream buffer
Leonardo Rossi picture Leonardo Rossi · Oct 31, 2013 · Viewed 81.7k times · Source

I have a relatively small file (some hundreds of kilobytes) that I want to be in memory for direct access for the entire execution of the code.

I don't know exactly the internals of Node.js, so I'm asking if a fs open is enough or I have to read all file and copy to a Buffer?

Answer

Golo Roden picture Golo Roden · Oct 31, 2013

Basically, you need to use the readFile or readFileSync function from the fs module. They return the complete content of the given file, but differ in their behavior (asynchronous versus synchronous).

If blocking Node.js (e.g. on startup of your application) is not an issue, you can go with the synchronized version, which is as easy as:

var fs = require('fs');

var data = fs.readFileSync('/etc/passwd');

If you need to go asynchronous, the code is like that:

var fs = require('fs');

fs.readFile('/etc/passwd', function (err, data ) {
  // ...
});

Please note that in either case you can give an options object as the second parameter, e.g. to specify the encoding to use. If you omit the encoding, the raw buffer is returned:

var fs = require('fs');

fs.readFile('/etc/passwd', { encoding: 'utf8' }, function (err, data ) {
  // ...
});

Valid encodings are utf8, ascii, utf16le, ucs2, base64 and hex. There is also a binary encoding, but it is deprecated and should not be used any longer. You can find more details on how to deal with encodings and buffers in the appropriate documentation.

Related questions

Converting a Buffer into a ReadableStream in Node.js

I'm fairly new to Buffers and ReadableStreams, so maybe this is a stupid question. I have a library that takes as input a ReadableStream, but my input is just a base64 format image. I could convert the data I have …

Read More
How to create streams from string in Node.Js?

I am using a library, ya-csv, that expects either a file or a stream as input, but I have a string. How do I convert that string into a stream in Node?

Read More
How do I read the contents of a Node.js stream into a string variable?

I'm hacking on a Node program that uses smtp-protocol to capture SMTP emails and act on the mail data. The library provides the mail data as a stream, and I don't know how to get that into a string. I'm …

Read More