How To Load-Distribution in RabbitMQ cluster?

amazon-ec2 amazon-web-services rabbitmq cluster-computing autoscaling
jayesh picture jayesh · May 5, 2012 · Viewed 7.8k times · Source

Hi I create three RabbitMQ servers running in cluster on EC2

I want to scale out RabbitMQ cluster base on CPU utilization but when I publish message only one server utilizes CPU and other RabbitMQ-server not utilize CPU

so how can i distribute the load across the RabbitMQ cluster

Answer

scvalex picture scvalex · May 5, 2012

RabbitMQ clusters are designed to improve scalability, but the system is not completely automatic.

When you declare a queue on a node in a cluster, the queue is only created on that one node. So, if you have one queue, regardless to which node you publish, the message will end up on the node where the queue resides.

To properly use RabbitMQ clusters, you need to make sure you do the following things:

  • have multiple queues distributed across the nodes, such that work is distributed somewhat evenly,
  • connect your clients to different nodes (otherwise, you might end up funneling all messages through one node), and
  • if you can, try to have publishers/consumers connect to the node which holds the queue they're using (in order to minimize message transfers within the cluster).

Alternatively, have a look at High Availability Queues. They're like normal queues, but the queue contents are mirrored across several nodes. So, in your case, you would publish to one node, RabbitMQ will mirror the publishes to the other node, and consumers will be able to connect to either node without worrying about bogging down the cluster with internal transfers.

Related questions

Permission denied (publickey) when SSH Access to Amazon EC2 instance

I want to use my Amazon ec2 instance but faced the following error: Permission denied (publickey). I have created my key pair and downloaded .pem file. Given: chmod 600 pem file. Then, this command ssh -i /home/kashif/serverkey.pem ubuntu@…

Read More
Trying to SSH into an Amazon Ec2 instance - permission error

This is probably a stupidly simple question to some :) I've created a new linux instance on Amazon EC2, and as part of that downloaded the .pem file to allow me to SSH in. When I tried to ssh with: ssh …

Read More
Change key pair for ec2 instance

How do I change the key pair for my ec2 instance in AWS management console? I can stop the instance, I can create new key pair, but I don't see any link to modify the instance's key pair.

Read More