Python interpteter can't find module in virtualenv, but pip sees it and won't install

Question

I'm trying to use the pytools module within the virtualenv created by Nervana for their Neon deep learning package, but can't seem to either find pytools or pip it. When I enter my virtualenv, I see this behavior:

me@ARL--M6800:~/Downloads/neon$ source .venv/bin/activate
(.venv) me@ARL--M6800:~/Downloads/neon$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

>>> import pytools
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  ImportError: No module named pytools
>>> 

>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.7/dist-packages', '/home/me/Downloads/neon',
 '/home/me/Downloads/neon/.venv/lib/python2.7',
 '/home/me/Downloads/neon/.venv/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/me/Downloads/neon/.venv/lib/python2.7/lib-tk',
 '/home/me/Downloads/neon/.venv/lib/python2.7/lib-old', 
 '/home/me/Downloads/neon/.venv/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7', 
 '/usr/lib/python2.7/plat-x86_64-linux-gnu', 
 '/usr/lib/python2.7/lib-tk', 
 '/home/me/Downloads/neon/.venv/local/lib/python2.7/site-packages', 
 '/home/me/Downloads/neon/.venv/lib/python2.7/site-packages']



(.venv) me@ARL--M6800:~/Downloads/neon$ pip install pytools 
Requirement already satisfied (use --upgrade to upgrade): pytools in     
 /usr/local/lib/python2.7/dist-packages/pytools-2016.1-py2.7.egg
Requirement already satisfied (use --upgrade to upgrade): 
 decorator>=3.2.0 in /usr/local/lib/python2.7/dist-packages (from pytools)
Requirement already satisfied (use --upgrade to upgrade): appdirs>=1.4.0 
  in /usr/local/lib/python2.7/dist-packages/appdirs-1.4.0-py2.7.egg (from 
  pytools)
Requirement already satisfied (use --upgrade to upgrade): six>=1.8.0 in 
 /usr/local/lib/python2.7/dist-packages (from pytools)
Requirement already satisfied (use --upgrade to upgrade): numpy>=1.6.0 in 
 /usr/local/lib/python2.7/dist-packages (from pytools)

So, I can't import pytools becauseit isn't on my sys.path. According to pip, it is installed in the /usr/local/lib/python2.7/dist-packages/pytools-2016.1-py2.7.egg directory, which leaves me with 3 questions:

First: Why can my virtualenv see my system-wide packages? I thought the default was not to see them. When I look at the Makefile used to create the virtualenv, I see this

  # where our installed python packages will live
  VIRTUALENV_DIR := .venv
  VIRTUALENV_EXE := virtualenv -p python2.7  # use pyvenv for python3 install
  ACTIVATE := $(VIRTUALENV_DIR)/bin/activate

which should give me default behavior.

Second: Why are there egg directories in my dist-packages dir? Doesn't this make it harder to find those modules? (Though apparently, the sys.path for my system environment python has been updated to search in the egg dir. When/How???)

Third: What is an efficient way of fixing things so that my virtualenv will have access to pytools?

(I would've numbered my list instead of First/Second/Third, but then the Makefile code I inserted didn't format well)

Answer 1

First of all, I don't know anything about Nervana's Neon.

I get the same sys.path when the virtualenv is created with root privileges:

dm@Z580:~$ sudo virtualenv test1 -p python2.7
[sudo] password for dm: 
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in test1/bin/python2.7
Also creating executable in test1/bin/python
Installing setuptools, pip, wheel...done.
dm@Z580:~$ source test1/bin/activate
(test1)dm@Z580:~$ which python
/home/dm/test1/bin/python
(test1)dm@Z580:~$ python
Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pprint
>>> pprint.pprint(sys.path)
['',
 '/home/dm/test1/lib/python2.7',
 '/home/dm/test1/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/dm/test1/lib/python2.7/lib-tk',
 '/home/dm/test1/lib/python2.7/lib-old',
 '/home/dm/test1/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/home/dm/test1/local/lib/python2.7/site-packages',
 '/home/dm/test1/lib/python2.7/site-packages']

I would remove the current virtualenv, recreate it without sudo and then install all requirements with the virtualenv's pip.

cd ~/Downloads
rm -rf neon/.venv/
virtualenv neon/.venv/ -p python2.7
source neon/.venv/bin/activate
pip install -r neon/requirements.txt