How to restart Celery gracefully without delaying tasks

django linux celery pid celeryd
nitwit picture nitwit · Mar 10, 2012 · Viewed 14.3k times · Source

We use Celery with our Django webapp to manage offline tasks; some of these tasks can run up to 120 seconds.

Whenever we make any code modifications, we need to restart Celery to have it reload the new Python code. Our current solution is to send a SIGTERM to the main Celery process (kill -s 15 `cat /var/run/celeryd.pid`), then to wait for it to die and restart it (python manage.py celeryd --pidfile=/var/run/celeryd.pid [...]).

Because of the long-running tasks, this usually means the shutdown will take a minute or two, during which no new tasks are processed, causing a noticeable delay to users currently on the site. I'm looking for a way to tell Celery to shutdown, but then immediately launch a new Celery instance to start running new tasks.

Things that didn't work:

  • Sending SIGHUP to the main process: this caused Celery to attempt to "restart," by doing a warm shutdown and then relaunching itself. Not only does this take a long time, it doesn't even work, because apparently the new process launches before the old one dies, so the new one complains ERROR: Pidfile (/var/run/celeryd.pid) already exists. Seems we're already running? (PID: 13214) and dies immediately. (This looks like a bug in Celery itself; I've let them know about it.)
  • Sending SIGTERM to the main process and then immediately launching a new instance: same issue with the Pidfile.
  • Disabling the Pidfile entirely: without it, we have no way of telling which of the 30 Celery process are the main process that needs to be sent a SIGTERM when we want it to do a warm shutdown. We also have no reliable way to check if the main process is still alive.

Answer

mher picture mher · Mar 10, 2012

celeryd has --autoreload option. If enabled, celery worker (main process) will detect changes in celery modules and restart all worker processes. In contrast to SIGHUP signal, autoreload restarts each process independently when the current executing task finishes. It means while one worker process is restarting the remaining processes can execute tasks.

http://celery.readthedocs.org/en/latest/userguide/workers.html#autoreloading

Related questions

how to use python2.7 pip instead of default pip

I just installed python 2.7 and also pip to the 2.7 site package. When I get the version with: pip -V It shows: pip 1.3.1 from /usr/lib/python2.6/site-packages (python 2.6) How do I use the 2.7 version of pip located at: /usr/local/…

Read More
Installing memcached for a django project

From the django docs: After installing Memcached itself, you'll need to install a memcached binding. There are several python memcached bindings available; the two most common are python-memcached and pylibmc. The pylibmc docs have their own requirements: -libmemcached 0.32 or later (…

Read More
Access Django devserver from another machine same network

I'm using django's builtin server to develop a site and I want other computers in the same network I'm on to be able to access the server using the local IP address. I have seen many posts about this and …

Read More