MongoDB as a queue service?

Question

I would love to hear more about real application experience witn MongoDB as a queue service, if you used MongoDB for this purpose could you share your thoughts, as well as the environment in which it was used?

Answer 1

I am using mongodb as a queue service for email sending. Soon it will work in following way:

1. When a new message comes I store it in the mongodb.
2. A background job then loads the message from mongodb via the atomic operation findAndModify and sets the flag Processing to true, so it does not process same message twice (because my background job runs multiple threads in parallel).
3. Once the email has been sent I remove the document from mongodb.
4. You can also keep count of the failures for each message and remove it after 3 failed attempts.

In general I use mongodb as a queue service only for one reason: because I need to send emails by specified schedule (each message contains information on what time it should be sent).

If you do not have any schedule and need to process message immediately, I suggest that you look into existing queue services, because they probably handle all cases that you may not see without a deeper understanding of message queues.

Update

When background job crashes during message processing you could do following:

1. Move this message to another, message queue errors collection or..

2. Increase processing attempts counter in a message and again assign status "New", to try process it again. Just make sure that background job is idempotent (can process same message multiple times and not corrupt data) and transactional (when job fails you must undone changes that was made. if any). When job fails after 5 attempts (config value) perform #1.

3. Once bug with message processing was fixed you could process it again once more by assigning "New" status and moving to the message queue, or just delete this message. It depends on business processes actually.