Delayed_job - Multiple parallel queues?
I am using delayed_job and moved to a new beefier server. So now I would like to run parallel jobs, as now I have the POWER!, but am confused on whether delayed_job can run multiple parallel queues?
This question suggested that there are named queues, but do these all run off the one table and are thus sequential?
At the bottom @Jesse Wolgamott suggests that you can create a table for each queue that will then run in parrallel.
Has anyone done this and can they point me to how it is done?
Answer
It is possible and I am doing it all the time. In our case we need multiple jobs for processing three different kinds of jobs say queue_a, queue_b, and queue_c. The system will make entries in the delayed_job table and the queue named appropriately.
Start multiple delayed jobs like
RAILS_ENV=production script/delayed_job -i first --queue=queue_a start
RAILS_ENV=production script/delayed_job -i second --queue=queue_a start
RAILS_ENV=production script/delayed_job -i third --queue=queue_b start
RAILS_ENV=production script/delayed_job -i fourth --queue=queue_c start
Each command will create a delayed_job, so there will be now 4 parallel jobs, two of them serving queue_a and one each for queue_b and queue_c. The key thing here is the identifier that is passed through the -i option which specifies the instance name, and we can start and stop jobs as required.
Another option is to use worker pools.
RAILS_ENV=production script/delayed_job --pool=tracking --pool=mailers,tasks:2 --pool=*:2 start
That command will start 1 worker for the tracking queue, 2 workers for the mailers and tasks queues, and 2 workers for any jobs.