Python dependency - hell no!

19th September, 2018

5 min read

Dependency and version hell

Python projects can very quickly run into issues with incompatible dependencies. This is because by default Python exposes all dependencies globally, rather than allowing you install only the dependencies you need for a particular project. Also, if you need to use different versions of Python for different projects, you will have a hard road ahead of you.

I find all these problems are solved using pyenv in combination with virtualenv. Other people have various solutions, this works well for me and is stable and easy to setup.

Homebrew installations

These instructions are designed for Mac and assume you use homebrew

pyenv

(From https://github.com/pyenv/pyenv#homebrew-on-mac-os-x)

brew update
brew install pyenv

Add pyenv init to bash profile

echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile

Restart the terminal

Virtualenv

(from https://github.com/pyenv/pyenv-virtualenv#installing-with-homebrew-for-os-x-users)

brew install pyenv-virtualenv

Now add pyenv virtualenv-init to your bash profile

echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile

Usage

To install a version of Python - that works with Tensorflow for instance:

pyenv install 3.5.0
# after you install it run rehash
pyenv rehash

You should run rehash after installing a new version of Python or when an installation provides binaries. This is meant to rebuild the softlinks that pyenv uses to provide separate Python environments.

You can now choose what version of Python your open terminal will use as follows

# set the global version of python
pyenv global 3.5.0
# set the version of python to use in this terminal session
pyenv shell 3.5.0

You can check what versions of Python are installed - the asterisk indicates the current global python version.

pyenv versions
# produces a list like...
#  *  3.5.0
#     2.7.14

This will solve your problem of which version of Python is available to you, but you will still have an issue with dependency conflicts - a problem that virtualenv will solve for us.

Virtualenv

This allows us to create a separate Python installation in which we can then install any dependencies we need to suit the needs of a particular project. Virtualenv provides tools to create an environment, make it active for a project / terminal session, deactivate it and uninstall the environment.

A new environment can be created using the following command, where 3.5.0 is the version of Python you want to use in the environment. That version of Python will need to have been already installed with pyenv.

The label someprojectname is any name that is used to identify the environment, and needs to be unique - the project name is a good choice.

pyenv virtualenv 3.5.0 someprojectname

When you specify a new environment virtualenv will copy Python, pip and a bunch of other core dependencies to the specified folder.

The current virtualenvs can be listed using

pyenv virtualenvs
# 3.5.0/envs/someprojectname (created from /Users/yourhomedir/.pyenv/versions/3.5.0)
# * someprojectname (created from /Users/yourhomedir/.pyenv/versions/3.5.0)

The virtual environments will also be listed in the versions - note that the listing is slightly different, excluding the "created from..." details

pyenv versions
#  *  3.5.0
#     3.5.0/envs/someprojectname
#     2.7.14

You need to activate a virtual environment, which means the terminal session will use the version of Python associated with the virtual environment, including the specific dependencies. Note, you do not need to reinstall the dependencies each time you activate the environment.

Activating a virtualenv is simple enough

pyenv activate someprojectname

This will activate the virtual environment for the specific terminal session. You can also use pyenv local to allow the virutalenv activate whenever you enter a folder.

pyenv local someprojectname

This will activate the someprojectname environment and drop a .python-version file in the current folder that contains the environment name. Now every time you enter the folder the environment will activate, leaving the folder will deactivate it.

If you activate the environment before entering the folder it will remain active when you leave the folder.

Deployment with pyenv and pyenv-virtualenv

NOTE Work In Progess The target server, or more likely Docker container, should have pyenv and pyenv-virtualenv installed. The environment variable PYENV_ROOT should be set to a location within the app folder

PYENV_ROOT=/opt/app/.pyenv

The various bash profile files should point to the same shims folder and include the usual pyenv init and pyenv virtualenv-init commands.

export PATH=~/.pyenv/shims:~/.pyenv/bin:$PATH

TL:DR - quick reference sheet

Assuming you have followed the instructions above and installed pyenv and pyenv-virtualenv properly:

To add another version of Python

pyenv install 3.6.1
pyenv rehash

List the installed versions (and environments)

pyenv versions

List the virtualenvs

pyenv virtualenvs

To create a virtualenv for a project

pyenv virtualenv 2.7.14 someprojectname

To activate a virtualenv

pyenv activate someprojectname

To activate the virtualenv when you enter a (project) folder

cd /path/to/your/project/folder/
pyenv local someprojectname

and deactivate that environment

pyenv deactivate

To remove a virtual env

Useful if you installed the wrong version of Python for some reason

pyenv uninstall someprojectname