Is it possible to move / merge messages between RabbitMQ queues?
I'm looking to know is it possible to move / merge messages from one queue to another. For example:
main-queue contains messages ['cat-1','cat-2','cat-3','cat-4','dog-1','dog-2','cat-5']
dog-queue contains messages ['dog-1, dog-2, dog-3, dog-4]
So the question is, (assuming both queues are on the same cluster, vhost) it possible to move messages from dog-queue to main-queue using rabbitmqctl ?
So at the end I'm looking to get something like:
Ideally:
main-queue : ['cat-1','cat-2','cat-3','cat-4','dog-1','dog-2','cat-5', dog-3, dog-4]
But this is ok too:
main-queue : ['cat-1','cat-2','cat-3','cat-4','dog-1','dog-2','cat-5', 'dog-1, dog-2, dog-3, dog-4]
Answer
What you are/were looking for is the 'shovel' plugin. The shovel plugin comes built into the core but you have to explicitly enable it. It's really easy to use as it does everything for you (no manually consuming/republishing to another queue).
Enable shovel plugin via cli:
sudo rabbitmq-plugins enable rabbitmq_shovel
If you manage RabbitMQ via GUI, install the shovel mgmt plugin too:
sudo rabbitmq-plugins enable rabbitmq_shovel_management
Login to the GUI and you will see Shovel Management under the Admin section. You can create shovels to move messages from any queue to another queue, even remotely hosted queues. You can delete the shovel when it's finished, or leave it there and it'll continually move the messages as they come in.
If you manage RabbitMQ via CLI, you can execute shovel directly from rabbitmqctl:
sudo rabbitmqctl set_parameter shovel cats-and-dogs \
'{"src-uri": "amqp://user:pass@host/vhost", "src-queue": "dog-queue", \
"dest-uri": "amqp://user:pass@host/vhost", "dest-queue": "main-queue"}'
Official plugin docs:
Shovel Plugin - https://www.rabbitmq.com/shovel.html
Creating Shovels - https://www.rabbitmq.com/shovel-dynamic.html