What is the proper way to work with shared modules in Python development?

python python-3.x distutils
Steve Sawyer picture Steve Sawyer · Jun 18, 2013 · Viewed 26.1k times · Source

I'm working toward adopting Python as part of my team's development tool suite. With the other languages/tools we use, we develop many reusable functions and classes that are specific to the work we do. This standardizes the way we do things and saves a lot of wheel re-inventing.

I can't seem to find any examples of how this is usually handled with Python. Right now I have a development folder on a local drive, with multiple project folders below that, and an additional "common" folder containing packages and modules with re-usable classes and functions. These "common" modules are imported by modules within multiple projects.

Development/
    Common/
        Package_a/
        Package_b/
    Project1/
        Package1_1/
        Package1_2/
    Project2/
        Package2_1/
        Package2_2/

In trying to learn how to distribute a Python application, it seems that there is an assumption that all referenced packages are below the top-level project folder, not collateral to it. The thought also occurred to me that perhaps the correct approach is to develop common/framework modules in a separate project, and once tested, deploy those to each developer's environment by installing to the site-packages folder. However, that also raises questions re distribution.

Can anyone shed light on this, or point me to a resource that discusses this issue?

Answer

robjohncox picture robjohncox · Jun 18, 2013

If you have common code that you want to share across multiple projects, it may be worth thinking about storing this code in a physically separate project, which is then imported as a dependency into your other projects. This is easily achieved if you host your common code project in github or bitbucket, where you can use pip to install it in any other project. This approach not only helps you to easily share common code across multiple projects, but it also helps protect you from inadvertently creating bad dependencies (i.e. those directed from your common code to your non common code).

The link below provides a good introduction to using pip and virtualenv to manage dependencies, definitely worth a read if you and your team are fairly new to working with python as this is a very common toolchain used for just this kind of problem:

http://dabapps.com/blog/introduction-to-pip-and-virtualenv-python/

And the link below shows you how to pull in dependencies from github using pip:

How to use Python Pip install software, to pull packages from Github?

Related questions

PyCrypto install error on Windows

I am trying to install PyCrypto 2.6 Library on my computer. But I keep getting the following error D:\Software\Python\package\pycrypto-2.6>python setup.py build running build running build_py running build_ext warning: GMP or MPIR library …

Read More
Distribute pre-compiled python extension module with distutils

Quick one today: I'm learning the in's and out's of Pythons distutils library, and I would like to include a python extension module (.pyd) with my package. I know of course that the recommended way is to have distutils compile …

Read More
Convert bytes to a string

I'm using this code to get standard output from an external program: >>> from subprocess import * >>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0] The communicate() method returns an array of bytes: >>> …

Read More