How to configure __main__.py, __init__.py, and __setup__.py for a basic package setup?

Question

Background:

I have a directory structure like so:

Package/
    setup.py
    src/
        __init__.py
        __main__.py 
        code.py

I want to be able to run the code in a lot of different ways.

1. pip install Package and then python and then from Package import *

2. python -m Package which should do the thing in __main__.py

3. python __main__.py which should also do the thing in __main__.py but this time, we assume you've downloaded source rather than pip installing.

Now I've gotten the first two to work, but with a messy setup:

setup.py:

setup(
    name='Package',
    packages=['Package'],
    package_dir={'Package': 'src'},
    ...
    entry_points={ 'console_scripts': ['Package = src.__main__:main' ] }

__init__.py:

from Package.code import .......

__main__.py:

from . import .......

What would make more sense to me would be in both cases to write

from code import ........

but that gives me import errors.

Question:

Is the way I have it really the only way?

And most importantly, how do I support the third use case? Right now, python __main__.py throws

File "__main__.py", line 10, in <module>
    from . import code
ImportError: cannot import name 'class defined in code.py'

Notes:

I have read

• https://chriswarrick.com/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/
• http://setuptools.readthedocs.io/en/latest/setuptools.html
• The many questions here that look like this one but don't answer my question above.

Answer 1

You have almost everything you need (even a bit more)! I'd go with the following setup:

code.py:

foo = 1

__init__.py:

from .code import foo

Doing a relative import here because __init__.py will be used when importing the whole package. Note that we explicitly mark the import as relative by using the .-syntax because this is required for Python 3 (and in Python 2 if you did from __future__ import absolute_import).

__main__.py:

from Package import foo

print('foo = ', foo)

This is the package's main script and so we use an absolute import statement. By doing so we assume that the package has been installed (or at least has been put on the path); and that is the way packages should be dealt with! You might think that this conflicts with your third use case but actually there is no reason not to pip install when dealing with a package. And it really isn't a big deal (especially when using a virtualenv)!

If your concern is to tinker with the source files and readily observe the changes by running the __main__.py file then you can simply install the package using the -e ("editable") switch: pip install -e . (assuming you are in directory Package). With your current directory structure, however, this won't work because the -e switch will place an egg-link to the directory containing the setup.py file; this directory does not contain a package named Package but src instead (I have a question about that).

Instead, if you follow the convention to name the root directory of a package's source after the package itself (that is Package for your example) then installing with -e is not a problem: Python does find the required package Package in the corresponding directory:

$ tree Package/
Package/
├── setup.py
└── Package   <-- Renamed "src" to "Package" because that's the package's name.
    ├── code.py
    ├── __init__.py
    └── __main__.py

This also lets you omit the extra definition of package_dir={'Package': 'src'} in setup.py.

A note about setup.py: For the three use cases which you've specified there is no need to define an entry point. That is you can skip the line entry_points={ 'console_scripts': ['Package = src.__main__:main' ] }. By shipping a __main__.py module python -m Package will readily execute the code in this module. You can also add an extra if-clause:

def main():
    print('foo = ', foo)

if __name__ == '__main__':
    main()

The entry point on the other hand lets you directly execute the code in __main__.main from the CLI; that is running $ Package will execute the corresponding code.

Recap

The bottom line is that I'd always use pip install when dealing with packages. And why not, especially if you've already created a setup.py file? If changes to the package are to be applied "in real-time" then you can install with the -e switch (this might require a renaming of the src folder, see above). So your third use case would read as "Download the source and pip install (-e) Package (within a virtualenv); then you can run python __main__.py".

---

Edit

Run __main__.py without pip install

If you don't want to install the package via pip but still be able to run the __main__.py script, I'd still go with the above setup. Then we need to make sure that the from Package import ... statement(s) are still succeeding and this can be achieved by extending the import path (note that the this requires the src directory to be renamed to the package's name!).

Modify PYTHONPATH

For Linux bash you can set the Pythonpath as follows:

export PYTHONPATH=$PYTHONPATH:/path/to/Package

Or if you're in the same directory as __main__.py:

export PYTHONPATH=$PYTHONPATH:`cd ..; pwd`

Of course there are different ways for different operating systems.

Extend the path in __main__.py

You (or rather your colleague) could add the following lines to the top of the script (before the from Package import ... statements):

import sys
sys.path.append('/path/to/Package')

Extend the path in sitecustomize.py

You can place a module named sitecustomize.py in the lib/python3.5/site-packages/ directory of your Python installation which contains the following lines:

import sys
sys.path.append('/path/to/Package')

Create a separate, top-level main.py script

So you'd have the following layout:

$ tree Package/
Package/
├── main.py   <-- Add this file.
├── setup.py
└── src
    ├── code.py
    ├── __init__.py
    └── __main__.py

where main.py contains

import src.__main__

Now __main__.py is treated as a part of the src package and the relative import will work. Instead of running python src/__main__.py you would run python main.py now.