celeryev Queue in RabbitMQ Becomes Very Large

rabbitmq celery django-celery
speedplane picture speedplane · Jul 22, 2013 · Viewed 8.7k times · Source

I am using celery on rabbitmq. I have been sending thousands of messages to the queue and they are being processed successfully and everything is working just fine. However, the number of messages in several rabbitmq queues are growing quite large (hundreds of thousands of items in the queue). The queues are named celeryev.[...] (see screenshot below). Is this appropriate behavior? What is the purpose of these queues and shouldn't they be regularly purged? Is there a way to purge them more regularly, I think they are taking up quite a bit of disk space.

screenshot of rabbitmq control

Answer

Balthazar Rouberol picture Balthazar Rouberol · Nov 25, 2014

You can use the CELERY_EVENT_QUEUE_TTL celery option (only working with amqp), that will set the message expiry time, after which it will be deleted from the queue.

Related questions

Examples of Django and Celery: Periodic Tasks

I have been fighting the Django/Celery documentation for a while now and need some help. I would like to be able to run Periodic Tasks using django-celery. I have seen around the internet (and the documentation) several different formats …

Read More
RabbitMQ set_permissions syntax noob

I've seen other posts on stackoverflow related in some ways to the ridiculous question I'm about to ask but none of them dumbed it down enough for me to get what I was looking for.. I've installed rabbitmq and it's …

Read More
Celery task schedule (Celery, Django and RabbitMQ)

I want to have a task that will execute every 5 minutes, but it will wait for last execution to finish and then start to count this 5 minutes. (This way I can also be sure that there is only one task …

Read More