Python: import module from another directory at the same level in project hierarchy

Question

I've seen all sorts of examples and other similar questions, but I can't seem to find an example that exactly matches my scenario. I feel like a total goon asking this because there are so many similar questions, but I just can't seem to get this working "correctly." Here is my project:

user_management  (package)
        |
        |------- __init__.py
        |
        |------- Modules/
        |           |
        |           |----- __init__.py
        |           |----- LDAPManager.py
        |           |----- PasswordManager.py
        |
        |------- Scripts/
        |           |
        |           |----- __init__.py
        |           |----- CreateUser.py
        |           |----- FindUser.py

If I move "CreateUser.py" to the main user_management directory, I can easily use: "import Modules.LDAPManager" to import LDAPManager.py --- this works. What I can't do (which I want to do), is keep CreateUser.py in the Scripts subfolder, and import LDAPManager.py. I was hoping to accomplish this by using "import user_management.Modules.LDAPManager.py". This doesn't work. In short, I can get Python files to easily look deeper in the hierarchy, but I can't get a Python script to reference up one directory and down into another.

Note that I am able to solve my problem using:

sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
import Modules.LDAPManager as LDAPManager

I've heard that this is bad practice and discouraged.

The files in Scripts are intended to be executed directly (is the init.py in Scripts even necessary?). I've read that in this case, I should be executing CreateUser.py with the -m flag. I've tried some variations on this and just can't seem to get CreateUser.py to recognize LDAPManager.py.

Answer 1

If I move CreateUser.py to the main user_management directory, I can easily use: import Modules.LDAPManager to import LDAPManager.py --- this works.

Please, don't. In this way the LDAPManager module used by CreateUser will not be the same as the one imported via other imports. This can create problems when you have some global state in the module or during pickling/unpickling. Avoid imports that work only because the module happens to be in the same directory.

When you have a package structure you should either:

• Use relative imports, i.e if the CreateUser.py is in Scripts/:

   from ..Modules import LDAPManager
  

  Note that this was (note the past tense) discouraged by PEP 8 only because old versions of python didn't support them very well, but this problem was solved years ago. The current version of PEP 8 does suggest them as an acceptable alternative to absolute imports. I actually like them inside packages.

• Use absolute imports using the whole package name(CreateUser.py in Scripts/):

   from user_management.Modules import LDAPManager
  

In order for the second one to work the package user_management should be installed inside the PYTHONPATH. During development you can configure the IDE so that this happens, without having to manually add calls to sys.path.append anywhere.

Also I find it odd that Scripts/ is a subpackage. Because in a real installation the user_management module would be installed under the site-packages found in the lib/ directory (whichever directory is used to install libraries in your OS), while the scripts should be installed under a bin/ directory (whichever contains executables for your OS).

In fact I believe Script/ shouldn't even be under user_management. It should be at the same level of user_management. In this way you do not have to use -m, but you simply have to make sure the package can be found (this again is a matter of configuring the IDE, installing the package correctly or using PYTHONPATH=. python Scripts/CreateUser.py to launch the scripts with the correct path).

---

In summary, the hierarchy I would use is:

user_management  (package)
        |
        |------- __init__.py
        |
        |------- Modules/
        |           |
        |           |----- __init__.py
        |           |----- LDAPManager.py
        |           |----- PasswordManager.py
        |

 Scripts/  (*not* a package)
        |  
        |----- CreateUser.py
        |----- FindUser.py

Then the code of CreateUser.py and FindUser.py should use absolute imports to import the modules:

from user_management.Modules import LDAPManager

During installation you make sure that user_management ends up somewhere in the PYTHONPATH, and the scripts inside the directory for executables so that they are able to find the modules. During development you either rely on IDE configuration, or you launch CreateUser.py adding the Scripts/ parent directory to the PYTHONPATH (I mean the directory that contains both user_management and Scripts):

PYTHONPATH=/the/parent/directory python Scripts/CreateUser.py

Or you can modify the PYTHONPATH globally so that you don't have to specify this each time. On unix OSes (linux, Mac OS X etc.) you can modify one of the shell scripts to define the PYTHONPATH external variable, on Windows you have to change the environmental variables settings.

---

Addendum I believe, if you are using python2, it's better to make sure to avoid implicit relative imports by putting:

from __future__ import absolute_import

at the top of your modules. In this way import X always means to import the toplevel module X and will never try to import the X.py file that's in the same directory (if that directory isn't in the PYTHONPATH). In this way the only way to do a relative import is to use the explicit syntax (the from . import X), which is better (explicit is better than implicit).

This will make sure you never happen to use the "bogus" implicit relative imports, since these would raise an ImportError clearly signalling that something is wrong. Otherwise you could use a module that's not what you think it is.