How do setuptools, distribute, and pip relate to one another?

Question

I've been teaching myself Python through the book "Learn Python The Hard Way" (2nd Edition). In exercise 46 it told me to read up on Pip, Distribute, and a few other packages.

The documentation for pip was clear enough. It allows me to install/uninstall, and upgrade packages. Reading the documentation for distribute, it basically seems to do the same thing:

Easily download, build, install, upgrade, and uninstall Python packages

What's the difference between pip and distribute, and how do they relate to one another?

Answer 1

[2014-10 TL;DR: distribute is dead, use pip, the new setuptools, and, for binary distributions, wheels. More below.]

---

[Original answer]

Distribute is was a fork of the older setuptools so nearly all comments that follow apply equally to Distribute and setuptools. Setuptools was an attempt to fill in a number of holes in the even older Python standard library package, Distutils. Setuptools added functions like automatic downloads of packages via a command-line interface, easy_install, and some level of dependency management. However, a segment of the Python community is of the opinion that setuptools is too intrusive and has too much behind-the-scenes magic for some of its features.

pip is a higher-level interface on top of setuptools or Distribute. It uses them to perform many of its functions but avoids some of their more controversial features, like zipped eggs. pip also provides features not available in setuptools, like an uninstall command and the ability to define fixed sets of requirements and reliably reproduce a set of packages. There is a more complete feature comparison here.

Why are there so many components (and there are more, like buildout)? Lots of reasons: solutions must work across all of the major platforms on which Python is supported (i.e. Unix-y, Windows, Mac OS X), so building and installation present a complex set of problems; like many open-source projects, Python is essentially all-volunteer and many developers just aren't all that interested in packaging and installation issues; there is a natural conservatism about adding major new unproven features to the standard library; differences in opinions, etc etc.

At the moment, there is a project underway to provide a replacement for Distutils and possibly for some of the higher-level add-ons. It is planned to be released in the Python 3.3 standard library as the packaging package and as an add-on for older versions of Python as Distutils2.

To summarize, the current relationship is:

pip -> [ setuptools | Distribute ] -> Distutils -> Python core
                                    |
    3rd party packages              |      included in Python
                                    |

---

UPDATE (2012-07): Prior to feature code cutoff for Python 3.3, it was decided that packaging was not quite ready yet for release in the standard library so it has been removed from the 3.3 release. Work will continue on Distutils2 which is available via PyPI and on what will be included in the standard library for Python 3.4.

---

UPDATE (2014-10): There have been further changes in the world of Python packaging since this answer was last updated.

• Most importantly, since mid-2013, the rift between setuptools and distribute has been healed and development activity has been merged into a new setuptools project. distribute is now deprecated and no longer maintained; use the new setuptools instead but don't use its easy_install as an installer.

• pip has become the de-facto and blessed installer tool (for Python packages not otherwise provided by your platform's package manager) either in- or outside of virtual environments (virtualenv or pyvenv).

• Instead of the old setuptools bdist eggs, wheels have become the blessed binary distribution format for Python packages.

• As of Python 3.4, a version of pip with wheel support is now shipped with the official python.org binary installers and source packages and it is anticipated that pip will also be included in the next maintenance release of Python 2.7 (2.7.9).

• Distutils2 and packaging are now dormant.

More details in the new Distributing Python Modules section of the Python 3 docs and the new Python Packaging User Guide.