RabbitMQ/AMQP - Best Practice Queue/Topic Design in a MicroService Architecture

Question

We are thinking of introducing a AMQP based approach for our microservice infrastructure (choreography). We do have several services, let's say the customer-service, user-service, article-service etc. We are planning to introduce RabbitMQ as our central Messaging System.

I'am looking for best practices for the design of the system regarding topics/queues etc. One option would be to create a message queue for every single event which can occur in our system, for example:

user-service.user.deleted
user-service.user.updated
user-service.user.created
...

I think it is not the right approach to create hundreds of message queues, isn't it?

I would like to use Spring and these nice annotations, so for example:

  @RabbitListener(queues="user-service.user.deleted")
  public void handleEvent(UserDeletedEvent event){...

Isn't it better to just have something like "user-service-notifications" as one queue and then send all notifications to that queue? I would still like to register listeners just to a subset of all events, so how to solve that?

My second question: If I want to listen on a queue which was not created before, I will get an exception in RabbitMQ. I know I can "declare" a queue with the AmqpAdmin, but should I do this for every queue of my hundreds in every single microservice, as it always can happen that the queue was not created so far?

Answer 1

I generally find it is best to have exchanges grouped by object type / exchange type combinations.

in you example of user events, you could do a number of different things depending on what your system needs.

in one scenario, it might make sense to have an exchange per event as you've listed. you could create the following exchanges

| exchange     | type   |
|-----------------------|
| user.deleted | fanout |
| user.created | fanout |
| user.updated | fanout |

this would fit the "pub/sub" pattern of broadcasting events to any listeners, with no concern for what is listening.

with this setup, any queue that you bind to any of these exchanges will receive all messages that are published to the exchange. this is great for pub/sub and some other scenarios, but it might not be what you want all the time since you won't be able to filter messages for specific consumers without creating a new exchange, queue and binding.

in another scenario, you might find that there are too many exchanges being created because there are too many events. you may also want to combine the exchange for user events and user commands. this could be done with a direct or topic exchange:

| exchange     | type   |
|-----------------------|
| user         | topic  |

With a setup like this, you can use routing keys to publish specific messages to specific queues. For example, you could publish user.event.created as a routing key and have it route with a specific queue for a specific consumer.

| exchange     | type   | routing key        | queue              |
|-----------------------------------------------------------------|
| user         | topic  | user.event.created | user-created-queue |
| user         | topic  | user.event.updated | user-updated-queue |
| user         | topic  | user.event.deleted | user-deleted-queue |
| user         | topic  | user.cmd.create    | user-create-queue  |

With this scenario, you end up with a single exchange and routing keys are used to distribute the message to the appropriate queue. notice that i also included a "create command" routing key and queue here. this illustrates how you could combine patterns through.

I would still like to register listeners just to a subset of all events, so how to solve that?

by using a fanout exchange, you would create queues and bindings for the specific events you want to listen to. each consumer would create it's own queue and binding.

by using a topic exchange, you could set up routing keys to send specific messages to the queue you want, including all events with a binding like user.events.#.

if you need specific messages to go to specific consumers, you do this through the routing and bindings.

ultimately, there is no right or wrong answer for which exchange type and configuration to use without knowing the specifics of each system's needs. you could use any exchange type for just about any purpose. there are tradeoffs with each one, and that's why each application will need to be examined closely to understand which one is correct.

as for declaring your queues. each message consumer should declare the queues and bindings it needs before trying to attach to it. this can be done when the application instance starts up, or you can wait until the queue is needed. again, this depends on what your application needs.

i know the answer i'm providing is rather vague and full of options, rather than real answers. there are not specific solid answers, though. it's all fuzzy logic, specific scenarios and looking at the system needs.

FWIW, I've written a small eBook that covers these topics from a rather unique perspective of telling stories. it addresses many of the questions you have, though sometimes indirectly.