Google App Engine: ImportError: No module named appengine.ext

Question

I am trying to write a test for my GAE programme which uses the datastore. Following Google's Documentation, I see that I should be adding the path to my SDK into my PYTHONPATH. I did this using:

import sys
sys.path.remove('/usr/local/lib/python2.7/dist-packages')    # Has a 'google' module, which I want to be sure isn't interfering.
sys.path.insert(1,'/home/olly/google-cloud-sdk/platform/google_appengine')
sys.path.insert(1, '/home/olly/google-cloud-sdk/platform/google_appengine/lib/yaml/lib')

Then when the file is run:

Traceback (most recent call last):
 File "myapp_tests.py", line 20, in <module>
    from google.appengine.ext import ndb
ImportError: No module named appengine.ext

I have installed the SDK in the location above, and looking in /home/olly/google-cloud-sdk/platform/google_appengine/ I found the google folder, which has an __init__.py in it, along with appengine. Basically, the folder structure looks good to me, with them all being named correctly and having __init__.py files.

In an interactive console, after running the commands above, I found that I could run:

import google

no problem, but when I tried

import google.appengine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named appengine

It was my understanding that having the __init__.py() files in the directories meant that they could be imported as above. I also did a sudo find / --name "google", and the only thing that showed up that is also in my PYTHONPATH was the /usr/local/lib/python2.7/dist-packages, which I explicitly removed, and also inserted the rest of my paths in front of anyway.

I tried using GAE's own method of:

import dev_appserver
dev_appserver.fix_sys_path()

which added a whole lot of paths to sys.path, but still didn't help me make it work.

I also found that when I add '/home/olly/Servers/google_appengine/google' to my paths, I can run:

import appengine.ext

but running:

from appengine.ext import ndb

causes:

Traceback (most recent call last):
  File "booking_function_tests.py", line 16, in <module>
     from appengine.ext import ndb
  File "/home/olly/Servers/google_appengine/google/appengine/ext/ndb/__init__.py", line 7, in <module>
     from tasklets import *
  File "/home/olly/Servers/google_appengine/google/appengine/ext/ndb/tasklets.py", line 69, in <module>
     from .google_imports import apiproxy_stub_map
  File "/home/olly/Servers/google_appengine/google/appengine/ext/ndb/google_imports.py"    , line 11, in <module>
     from google3.storage.onestore.v3 import entity_pb
ImportError: No module named google3.storage.onestore.v3

Am I missing something really obvious? How should I go about importing ndb?

EDIT: I'm running the latest SDK (1.9.34), but I have the following code in my google_imports.py:

try:
  from google.appengine.datastore import entity_pb
  normal_environment = True
except ImportError:
  try:
    from google3.storage.onestore.v3 import entity_pb
    normal_environment = False
  except ImportError:
    # If we are running locally but outside the context of App Engine.
    try:
      set_appengine_imports()
      from google.appengine.datastore import entity_pb
      normal_environment = True
    except ImportError:
      raise ImportError('Unable to find the App Engine SDK. '
                        'Did you remember to set the "GAE" environment '
                        'variable to be the path to the App Engine SDK?')

Also, google.__path__ gives me the '/usr/local/lib/python2.7/dist-packages' path which I thought I removed earlier. Here is an excerpt of how I'm removing it:

import sys
sys.path.insert(1, '/home/olly/Servers/google_appengine')
sys.path.insert(1, '/home/olly/Servers/google_appengine/lib/yaml/lib')
sys.path.remove('/usr/local/lib/python2.7/dist-packages')

import google
print google.__path__
print sys.path


['/usr/local/lib/python2.7/dist-packages/google']
['/home/olly/Servers/google_appengine/myapp', '/home/olly/Servers/google_appengine/lib/yaml/lib', '/home/olly/Servers/google_appengine/google', '/home/olly/Servers/google_appengine', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PILcompat', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode']

So my sys.path is updated, but import google seems to still be importing from the no-longer-there path, which would be the crux of my problem I guess. Do I need to reload the path or something?

Answer 1

I run into these problems a lot less by always running inside a virtualenv.

I agree with snakecharmerrb you should get print google.__file__ or google.__path_ to figure out exactly what you're importing.

This snippet might also solve your problem:

import google

gae_dir = google.__path__.append('/path/to/appengine_sdk//google_appengine/google')
sys.path.insert(0, gae_dir) # might not be necessary

import google.appengine # now it's on your import path`