Celery beat not picking up periodic tasks

python django celery celerybeat
Rivadiz picture Rivadiz · Oct 24, 2017 · Viewed 15.8k times · Source

I am trying to get started with celery, but I can't get my task up and running. I have installed django-celery-beat and celery4.

My settings file.

Installed apps (with celery packages)

...
'django_celery_beat',
'django_celery_results'

the celery configuration

CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_RESULT_BACKEND = 'redis://localhost:6379'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

celery.py

from __future__ import absolute_import, unicode_literals
import os
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'sandbox.settings')

app = Celery('sandbox')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

and my simple task, which i configured to run through the admin panel of django celery beat.

from __future__ import absolute_import, unicode_literals
from sandbox.celery import app


@app.task()
def try_celery():
    print "Trying out Celery"

I am trying to run this task as a periodic task (beat) with cron tab as */2 * * * *

The log I am getting is,

$ celery -A sandbox worker --loglevel=debug

[2017-10-24 14:28:02,999: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-10-24 14:28:03,001: DEBUG/MainProcess] | Worker: Building graph...
[2017-10-24 14:28:03,002: DEBUG/MainProcess] | Worker: New boot order: {Beat, Timer, Hub, Pool, Autoscaler, StateDB, Consumer}
[2017-10-24 14:28:03,017: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-10-24 14:28:03,017: DEBUG/MainProcess] | Consumer: Building graph...
[2017-10-24 14:28:03,038: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Agent, Gossip, Heart, event loop}

 -------------- celery@mypc v4.1.0 (latentcall)
---- **** ----- 
--- * ***  * -- Linux-4.9.0-kali3-amd64-x86_64-with-Kali-kali-rolling-kali-rolling 2017-10-24 14:28:03
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         sandbox:0x7fe519d38610
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     redis://localhost:6379/
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery


[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap
  . sandbox.applications.cats.try_celery
  . sandbox.celery.debug_task

[2017-10-24 14:28:03,053: DEBUG/MainProcess] | Worker: Starting Hub
[2017-10-24 14:28:03,053: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,053: DEBUG/MainProcess] | Worker: Starting Pool
[2017-10-24 14:28:03,197: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,198: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-10-24 14:28:03,199: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-10-24 14:28:03,216: INFO/MainProcess] Connected to redis://localhost:6379/0
[2017-10-24 14:28:03,217: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,217: DEBUG/MainProcess] | Consumer: Starting Events
[2017-10-24 14:28:03,228: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:03,229: DEBUG/MainProcess] | Consumer: Starting Mingle
[2017-10-24 14:28:03,229: INFO/MainProcess] mingle: searching for neighbors
[2017-10-24 14:28:04,255: INFO/MainProcess] mingle: all alone
[2017-10-24 14:28:04,256: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,256: DEBUG/MainProcess] | Consumer: Starting Tasks
[2017-10-24 14:28:04,273: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,274: DEBUG/MainProcess] | Consumer: Starting Control
[2017-10-24 14:28:04,277: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,277: DEBUG/MainProcess] | Consumer: Starting Gossip
[2017-10-24 14:28:04,281: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,282: DEBUG/MainProcess] | Consumer: Starting Heart
[2017-10-24 14:28:04,284: DEBUG/MainProcess] ^-- substep ok
[2017-10-24 14:28:04,284: DEBUG/MainProcess] | Consumer: Starting event loop
[2017-10-24 14:28:04,285: DEBUG/MainProcess] | Worker: Hub.register Pool...
[2017-10-24 14:28:04,294: WARNING/MainProcess] /home/alexd/.virtualenvs/skate/local/lib/python2.7/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
  warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2017-10-24 14:28:04,294: INFO/MainProcess] celery@mypc ready.
[2017-10-24 14:28:04,295: DEBUG/MainProcess] basic.qos: prefetch_count->8

why my tasks are not working?

Update

log of running celery as beat

$ celery -A sandbox beat --loglevel=debug
celery beat v4.1.0 (latentcall) is starting.
Stale pidfile exists - Removing it.
__    -    ... __   -        _
LocalTime -> 2017-10-24 15:07:20
Configuration ->
    . broker -> redis://localhost:6379/0
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> celery.beat.PersistentScheduler
    . db -> celerybeat-schedule
    . logfile -> [stderr]@%DEBUG
    . maxinterval -> 5.00 minutes (300s)
[2017-10-24 15:07:20,216: DEBUG/MainProcess] Setting default socket timeout to 30
[2017-10-24 15:07:20,217: INFO/MainProcess] beat: Starting...
[2017-10-24 15:07:20,372: DEBUG/MainProcess] Current schedule:

[2017-10-24 15:07:20,373: DEBUG/MainProcess] beat: Ticking with max interval->5.00 minutes
[2017-10-24 15:07:20,373: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.
[2017-10-24 15:15:43,232: DEBUG/MainProcess] beat: Synchronizing schedule...
[2017-10-24 15:15:43,245: DEBUG/MainProcess] beat: Waking up in 5.00 minutes.

creation of task enter image description here this is how the task is created as a periodic task in the admin.

ps. I changed to interval option, may be some problems with my cron tab? anyway still not working.

Answer

sytech picture sytech · Oct 24, 2017

The root cause, in this case, is that the beat scheduler needs to be started with the appropriate arguments. You supplied the following command:

$ celery -A sandbox worker --loglevel=debug

However, to start celery with a beat schedule, (as opposed to a regular celery worker) you must specify beat rather than worker. Moreover, when using the django_celery_beat extension, it is necessary to use the Database scheduler django_celery_beat.schedulers:DatabaseScheduler rather than the default scheduler celery.beat.PersistentScheduler.

So the corrected command would be:

$ celery -A sandbox beat --loglevel=debug --scheduler django_celery_beat.schedulers:DatabaseScheduler

Supporting documentation

Related questions

Setting up periodic tasks in Celery (celerybeat) dynamically using add_periodic_task

I'm using Celery 4.0.1 with Django 1.10 and I have troubles scheduling tasks (running a task works fine). Here is the celery configuration: os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings') app = Celery('myapp') app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) app.…

Read More
Cancel an already executing task with Celery?

I have been reading the doc and searching but cannot seem to find a straight answer: Can you cancel an already executing task? (as in the task has started, takes a while, and half way through it needs to be …

Read More
Python SSL connection "EOF occurred in violation of protocol"

I'm using Django Celery task to connect to Facebook Graph API with requests lib using Gevent. Issue I'm constantly running at is that every now and then I get EOF occurred in violation of protocol exception. I've searched around and …

Read More