Background processes in Node.js
What is a good aproach to handle background processes in a NodeJS application?
Scenario: After a user posts something to an app I want to crunch the data, request additional data from external resources, etc. All of this is quite time consuming, so I want it out of the req/res loop. Ideal would be to just have a queue of jobs where you can quickly dump a job on and a daemon or task runner will always take the oldest one and process it.
In RoR I would have done it with something like Delayed Job. What is the Node equivalent of this API?
Answer
If you want something lightweight, that runs in the same process as the server, I highly recommend Bull. It has a simple API that allows for a fine grained control over your queues.
If you're looking for something that runs as a standalone worker process, perhaps look into Kue. It can run as a RESTful API server, and even has several front-end apps written for it.
If you're familiar with Ruby's Resque, there is a node implementation called Node-resque
Bull, Kue and Node-resque are all backed by Redis, which is ubiquitous among Node.js worker queues. All 3 would be able to do what RoR's DelayedJob does, it's matter of specific features that you want, and your API preferences.