How can I lock a file while writing to it asynchronously

Question

I've got two node threads running, one watching a directory for file consumption and another that is responsible for writing files to given directories.

Typically they won't be operating on the same directory, but for an edge case I'm working on they will be.

It appears that the consuming app is grabbing the files before they are fully written, resulting in corrupt files.

Is there a way I can lock the file until the writing is complete? I've looked into the lockfile module, but unfortunately I don't believe it will work for this particular application.

=====

The full code is far more than makes sense to put here, but the gist of it is this:

1. App spins off the watchers and listener

Listener:

• listen for file being added to db, export it using fs.writeFile

Watcher:

• watcher uses chokidar to track added files in each watched directory
• when found fs.access is called to ensure we have access to the file
  • fs.access seems to be unfazed by file being written
• the file is consumed via fs.createReadStream and then sent to server
  • filestream is necessary as we need the file hash

In this case the file is exported to the watched directory and then reimported by the watch process.

Answer 1

I'd use proper-lockfile for this. You can specify an amount of retries or use a retry config object to use an exponential backoff strategy. That way you can handle situations where two processes need to modify the same file at the same time.

Here's a simple example with some retry options:

const lockfile = require('proper-lockfile');
const Promise = require('bluebird');
const fs = require('fs-extra');
const crypto = require('crypto'); // random buffer contents

const retryOptions = {
    retries: {
        retries: 5,
        factor: 3,
        minTimeout: 1 * 1000,
        maxTimeout: 60 * 1000,
        randomize: true,
    }
};

let file;
let cleanup;
Promise.try(() => {
    file = '/var/tmp/file.txt';
    return fs.ensureFile(file); // fs-extra creates file if needed
}).then(() => {
    return lockfile.lock(file, retryOptions);
}).then(release => {
    cleanup = release;

    let buffer = crypto.randomBytes(4);
    let stream = fs.createWriteStream(file, {flags: 'a', encoding: 'binary'});
    stream.write(buffer);
    stream.end();

    return new Promise(function (resolve, reject) {
        stream.on('finish', () => resolve());
        stream.on('error', (err) => reject(err));
    });
}).then(() => {
    console.log('Finished!');
}).catch((err) => {
    console.error(err);
}).finally(() => {
    cleanup && cleanup();
});