Sharing files between microservices

Question

I'm trying to move a project from its current monolithic state to microservices architecture. The project is in Node.js, so I've started looking into Seneca.js, especially with its seneca-mesh module. Moving image manipulation (crop, resize, etc.) into a microservice seemed the most sensible first step, since it drastically slows down my application now.

When the application is monolithic, there is no problem in passing certain files into file-manipulation logic — just read it from local storage disk. With microsevices, however, if we keep in mind scalability, it becomes more difficult. Of course, I could build an image manipulation microservice, scale it up within the same host machine, and share directories I need between it, so they, too, can read from a local disk.

What if I want a truly scalable microservice, that can be run and scaled on different machines with different IP-adresses that don't share the same filesystem? I thought that maybe I could take advantage of Node's streaming API and send these files back and forth via HTTP or TCP or sockets or you name it.

As far as I've learned, Seneca.js cannot do it the right way. Of course, I could send a file from the main app to image manipulation service via Seneca.js like so:

fs.createReadStream('/files/hello.jpg')
  .on('data', function(data) {
    seneca.act({ role: 'file', cmd: 'chunk', data: data }, cb);
  })
  .on('end', function(err) {
    seneca.act({ role: 'file', cmd: 'end' });
  })
  .on('error', function(err) {
    seneca.act({ role: 'test', cmd: 'error' });
  });

And receive it in chunks:

seneca.add({ role: 'file', cmd: 'chunk' }, writeToFileCb);
seneca.add({ role: 'file', cmd: 'end' }, endFileWriteCb);

But this approach seems ugly and wheel-reinventive.

Another way would be to come up with some HTTP server and send files as multipart/form-data or application/octet-stream, like so:

fs.createReadStream('file.json')
  .pipe(request.post('http://image-manipulator'))

But this means reinventing the framework for microservice communication. All in all, I ask for advice on file sharing between distributed microservices and possible frameworks for this.

Answer 1

If you are approaching a microservice architecture, you should think of a microservice for managing files! Don't stream files around if you are microservice environment. For example you might create a FileManagerService with an API exposed for with CRUD implementation and only use seneca act/add for serving the important data ... file-url, size, etc..