Feature comparison between npm, pip, pipenv and poetry package managers

Question

How do the main features of npm compare with pip, pipenv and poetry package managers? And how do I use those features of pipenv or poetry?

This could primarily help someone transitioning from being JavaScript developer to a python developer.

I've never used used rubygems but it may also be useful to compare the main features of that to the main python package mangers. To help anyone coming from a ruby background.

Answer 1

Overview

Feature \ Package Manager	npm	pip	pipenv	poetry
Access to main repo (i.e. Pypi/npm)	✓	✓	✓	✓
Record top level dependencies	✓	✗	Pipfile	pyproject.toml
Record development dependencies	✓	✗	Pipfile	pyproject.toml
Lock versions of all dependencies	✓	✓	Pipfile.lock	poetry.lock
Switch between interpreter versions	nvm	✗	✗	✓
Direct publishing	✓	✗	✓*	✓
Run scripts	✓	✗	Pipfile	✗
Editable local packages	✓	✓	✓	✓
Integration with Intellij	✓	✓	partial	✗

• Directly publishing with pipenv is possible if using a 3rd party dev dependency, and a script to tie it all together, see below.

Disclaimer: I only have experiences of these packages managers on 'NIX systems (OSX in particular), pipenv markets itself as treating Windows is a first class citizen, I'm not sure how this works without pyenv, which is not available on Windows as far as I am aware.

Basic Usage

pipenv:

To get the most out of pipenv, pyenv should be installed. pipenv will be able to detect and use any version of python installed with pyenv, even if it is not activated. For example if a Pipfile has listed python 3.4 as a requirement: to successfully run pipenv install, pyenv install 3.4.0 should be run first.

To create a new Pipfile and venv (using python 3.7.x):

>>> pipenv --python 3.7

Or to install dependencies from an existing Pipfile.lock use the command below. This command can also be used to create a Pipfile and venv (defaulting to the newest available python version).

>>> pipenv install

To run commands within the created venv:

>>> pipenv run <script or command> 

e.g

>>> pipenv run python main.py

poetry

Poetry still uses pyenv but in a different way: The version of python you wish to use must be activated before calling poetry install or poetry run.

A pyproject.toml can be created using:

>>> poetry init

or a full directory structure can be created using:

>>> poetry new <dir>

Before we can install we must activate a version of python that matches what is specified in the pyproject.toml file.

>>> pyenv global <python version specified in pyproject.toml>

Now we are able to create the venv using the command below, if a poetry.lock file is present it will install all the dependencies listed in it.

>>> poetry install

To run commands within the created venv:

>>> poetry run <command>

If we change the global python version using pyenv we will no longer be able to run commands in the created venv. There is an exception to this if we use a locally created venv, see below.

Running your code using different python versions

Sometimes it's good to check that your code will work on both python 3.7 and python 3.4. This is not something we can take for granted.

pipenv

Only possible by deleting the venv in recreating it using a different python version:

>>> rm -rf <path to venv>
>>> pipenv --python <different python version>

A warning will likely be displayed that the python version of the venv does not match the one specified in the Pipfile, but as far as I can tell it is only a warning.

https://github.com/pypa/pipenv/issues/1071

poetry

Poetry is much better suited to this use case: Multiple venvs can be created side by side. To create and use a new venv switch python versions using pyenv then create a new venv.

>>> pyenv global <different python version>
>>> poetry install

An error will be thrown if the python version does not match the one specified in the pyproject.toml however a range of python versions can be specified using semver versioning.

Local venv

I prefer my venv to be installed in a .venv folder local to my project, this is similar to how npm works and allows me to delete the folder and reinstall if anything strange happens or if (in the case of pipenv) I want to easily change which version of python I'm using.

pipenv

To enable this feature: set the following environment variable.

>>> export PIPENV_VENV_IN_PROJECT="enabled"

poetry

This feature can be enabled using the following command:

>>> poetry config settings.virtualenvs.in-project true

But beware that this will change the behavior of poetry, it will no longer be possible to use quickly switch between different versions of python: Even if the python version is switched using pyenv all commands run using poetry run will use the venv (and its associated python version) that resides in the local directory.

https://github.com/sdispater/poetry/issues/108

Installing Packages

pipenv

Packages are easily installed and automatically added to the Pipfile and Pipfile.lock files using:

>>> pipenv install [--dev] <package name>

The --dev flag indicates a development dependency. Development dependencies will not be installed by default when using pipenv install.

Local packages can also be installed, allowing you to work on them and see your changes immediately:

>>> pipenv install -e <path to local package>

poetry

Packages are easily installed and automatically added to the pyproject.toml and poetry.lock files using:

>>> poetry add [--dev] <package name>

The --dev flag indicates a development dependency, Development dependencies will not be installed by default when using poetry install or added to the package when publishing.

Local packages can also be installed, allowing you to work on them and see your changes immediately:

>>> poetry add --path <path to local package> <name of package>

Not sure why the name of the package is needed, as it should already be defined by the local package. Also the author seems unconvinced about linking local packages in general (https://github.com/sdispater/poetry/issues/34) so this feature may get forgotten about over time.

Running Scripts

To be clear, I'm referring to what npm calls scripts, which is different to the scripts specified inside a setup.py file.

When developing it is sometimes useful to set up shortcuts for commands that are difficult to remember, for example the command for running every test file in a directory is:

>>> python -m unittest discover -s <test_folder> -p '*_test.py'

It is much more convenient to have a shortcut to these sorts of commands.

pipenv

This feature is supported: put the following into the Pipfile:

[scripts]
    test = "pipenv run python -m unittest discover -s tests -p '*_test.py'"

poetry

Not supported, and unlikely to be added in the future: https://github.com/sdispater/poetry/pull/591#issuecomment-504762152

Publishing to PyPi

It would be preferable to be able to publish to PyPi without crafting an additional setup.py file, this would be possible if all the information needed to publish the package was contained within the package management files.

pipenv

As far as I can tell this is where pipenv gets its bad reputation. setup.py files are still needed to publish to PyPi and no, they are not auto-populated with the dependencies from the Pipfile.

The recommended approach is to either copy the dependencies over manually when publishing, or to get the Pipfile to install the dependencies listed in the setup.py files, however, the setup.py is not automatically updated when running pipenv install <package name>.

If you really want your Pipfile to depend on your setup.py file, this is how it's done:

>>> pipenv install '-e .'

https://github.com/pypa/pipenv/issues/2805, https://realpython.com/pipenv-guide/#yes-i-need-to-distribute-my-code-as-a-package, https://github.com/pypa/pipenv/issues/209

Yuck!

---

So ideally we want to derive a setup.py file from the Pipfile:

I found two existing packages that claim to do this:

1. https://pypi.org/project/pipenv-tools/ - But it hasn't been updated in two years, there's no code in the src directory and I couldn't get it to work.

2. https://pypi.org/project/pipenv-setup/ - But it syncs the Pipfile.lock instead of the Pipfile, this is an anti-pattern. The lock file is meant for creating a reproducible environment, it is overly restrictive (e.g. by not allowing updates to dependencies) to be used for setup.py. For this reason I didn't even try using it.

---

My Solution:

I quickly wrote a package that generates an install_requires.py file that can be imported in a setup.py file: https://pypi.org/project/pipenv2setup/ (it is untested on Windows).

For an example of how to use the package when publishing pipenv projects, see this github repo:

https://github.com/alanbacon/pipenvExample

poetry

Publishing your package using poetry is really easy, you do not need a setup.py file at all. Simply run:

>>> poetry publish [--build] [--username <username>] [--password <password>]

The published package can be installed using pip not just by other instances of poetry.

For information about how to migrate from using a setup.py to purely a pyproject.toml file, see here: https://johnfraney.ca/posts/2019/05/28/create-publish-python-package-poetry/

IntelliJ or Pycharm integration

pipenv

Pycharm is able detect the venv by using the Piplock files, however adding new packages using the Pycharm interface will not modify the Piplock files.

poetry

At the time of writing Pycharm does not seem to be aware of any poetry virtual environments or appear to parse the pyproject.toml file in any way.

Other Points about poetry

Semver

In poetry you must specify versions of python and packages using semver (have to use ~ and ^, not >= or <) https://nodesource.com/blog/semver-tilde-and-caret.

Buggyness

poetry runs using python but doesn't work with older versions of python. So to develop for an older version of python: some commands have to be run with pyenv set to >3.6, but then the pyenv needs to be switched back to the older version to create the venv. It also appears that venvs have to be greater than 3.5. https://github.com/sdispater/poetry/issues/1223

Also not sure about windows compatibility for poetry.

Conclusion

To me the main differences between the poetry and pipenv lie in their usage of pyenv and their abilities (or lack of) to publish straight to PyPi. There is also the lack of scripts in poetry which I personally find frustrating.

I find that when using poetry there is a lot more switching between python environments using pyenv. Although currently this can be mitigated by using a locally install venv. I know this limits my ability to quickly test my code in different python environments, but there are other tools such as tox to do that.

Publishing to PyPi using poetry is so easy, I explained it in one line (above). Publishing to PyPi with pipenv is a minefield, to explain it I had to link to an entire git repo (above).