Django, ImportError: cannot import name Celery, possible circular import?

Question

I went through this example here:

http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

All my tasks are in files called tasks.py.

After updating celery and adding the file from the example django is throwing the following error, no matter what I try:

ImportError: cannot import name Celery

Is the problem possibly caused by the following?

app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

Because it goes through all tasks.py files which all have the following import.

from cloud.celery import app

cloud/celery.py:

from __future__ import absolute_import

import os, sys
from celery import Celery
from celery.schedules import crontab
from django.conf import settings

BROKER_URL = 'redis://:PASSWORD@localhost'

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'cloud.settings')
app = Celery('cloud', broker=BROKER_URL)
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')

if "test" in sys.argv:
    app.conf.update(
        CELERY_ALWAYS_EAGER=True,
    )
    print >> sys.stderr, 'CELERY_ALWAYS_EAGER = True'

CELERYBEAT_SCHEDULE = {
    'test_rabbit_running': {
        "task": "retail.tasks.test_rabbit_running",
        "schedule": 3600, #every hour
    },
    [..]

app.conf.update(
    CELERYBEAT_SCHEDULE=CELERYBEAT_SCHEDULE
)

retail/tasks.py:

from cloud.celery import app

import logging
from celery.utils.log import get_task_logger
logger = get_task_logger('tasks')
logger.setLevel(logging.DEBUG)

@app.task
def test_rabbit_running():
    import datetime
    utcnow = datetime.datetime.now()
    logger.info('CELERY RUNNING')

The error happens, when I try to access a url that is not valid, like /foobar.

Here is the full traceback:

Traceback (most recent call last):
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 126, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 255, in __call__
    response = self.get_response(request)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 178, in get_response
    response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 220, in handle_uncaught_exception
    if resolver.urlconf_module is None:
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/core/urlresolvers.py", line 342, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/opt/src/slicephone/cloud/cloud/urls.py", line 52, in 
    urlpatterns +=  patterns('', url(r'^search/', include('search.urls')))
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/conf/urls/__init__.py", line 25, in include
    urlconf_module = import_module(urlconf_module)
  File "/opt/virtenvs/django_slice/local/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)
  File "/opt/src/slicephone/cloud/search/urls.py", line 5, in 
    from handlers import SearchHandler
  File "/opt/src/slicephone/cloud/search/handlers.py", line 15, in 
    from places import handlers as placeshandler
  File "/opt/src/slicephone/cloud/places/handlers.py", line 23, in 
    import api as placesapi
  File "/opt/src/slicephone/cloud/places/api.py", line 9, in 
    from djapi import *
  File "/opt/src/slicephone/cloud/places/djapi.py", line 26, in 
    from tasks import add_single_place, add_multiple_places
  File "/opt/src/slicephone/cloud/places/tasks.py", line 2, in 
    from cloud.celery import app
  File "/opt/src/slicephone/cloud/cloud/celery.py", line 4, in 
    from celery import Celery
  File "/opt/src/slicephone/cloud/cloud/celery.py", line 4, in 
    from celery import Celery
ImportError: cannot import name Celery

Answer 1

Adding the following lines to cloud/celery.py:

import celery
print celery.__file__

gave me the file itself and not the celery module from the library. After renaming celery.py to celeryapp.py and adjusting the imports all errors were gone.

Note:

That leads to a change in starting the worker:

celery worker --app=cloud.celeryapp:app

For those running celery==3.1.2 and getting this error:

TypeError: unpack_from() argument 1 must be string or read-only buffer, not memoryview

Apply the patch mentioned here: https://github.com/celery/celery/issues/1637