Python dependency - hell no!
19th September, 2018
5 min read
Dependency and version hell
Python projects can very quickly run into issues with incompatible dependencies. This is because by default Python exposes all dependencies globally, rather than allowing you install only the dependencies you need for a particular project. Also, if you need to use different versions of Python for different projects, you will have a hard road ahead of you.
I find all these problems are solved using pyenv in combination with virtualenv. Other people have various solutions, this works well for me and is stable and easy to setup.
These instructions are designed for Mac and assume you use homebrew
brew update brew install pyenv
pyenv init to bash profile
echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n eval "$(pyenv init -)"\nfi' >> ~/.bash_profile
Restart the terminal
brew install pyenv-virtualenv
pyenv virtualenv-init to your bash profile
echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.bash_profile
To install a version of Python - that works with Tensorflow for instance:
pyenv install 3.5.0 # after you install it run rehash pyenv rehash
You should run
rehash after installing a new version of Python or when an installation provides binaries. This is meant to rebuild the softlinks that pyenv uses to provide separate Python environments.
You can now choose what version of Python your open terminal will use as follows
# set the global version of python pyenv global 3.5.0 # set the version of python to use in this terminal session pyenv shell 3.5.0
You can check what versions of Python are installed - the asterisk indicates the current global python version.
pyenv versions # produces a list like... # * 3.5.0 # 2.7.14
This will solve your problem of which version of Python is available to you, but you will still have an issue with dependency conflicts - a problem that virtualenv will solve for us.
This allows us to create a separate Python installation in which we can then install any dependencies we need to suit the needs of a particular project. Virtualenv provides tools to create an environment, make it active for a project / terminal session, deactivate it and uninstall the environment.
A new environment can be created using the following command, where
3.5.0 is the version of Python you want to use in the environment. That version of Python will need to have been already installed with pyenv.
someprojectname is any name that is used to identify the environment, and needs to be unique - the project name is a good choice.
pyenv virtualenv 3.5.0 someprojectname
When you specify a new environment
virtualenv will copy Python, pip and a bunch of other core dependencies to the specified folder.
The current virtualenvs can be listed using
pyenv virtualenvs # 3.5.0/envs/someprojectname (created from /Users/yourhomedir/.pyenv/versions/3.5.0) # * someprojectname (created from /Users/yourhomedir/.pyenv/versions/3.5.0)
The virtual environments will also be listed in the versions - note that the listing is slightly different, excluding the "created from..." details
pyenv versions # * 3.5.0 # 3.5.0/envs/someprojectname # 2.7.14
You need to activate a virtual environment, which means the terminal session will use the version of Python associated with the virtual environment, including the specific dependencies. Note, you do not need to reinstall the dependencies each time you activate the environment.
Activating a virtualenv is simple enough
pyenv activate someprojectname
This will activate the virtual environment for the specific terminal session. You can also use
pyenv local to allow the virutalenv activate whenever you enter a folder.
pyenv local someprojectname
This will activate the
someprojectname environment and drop a
.python-version file in the current folder that contains the environment name. Now every time you enter the folder the environment will activate, leaving the folder will deactivate it.
If you activate the environment before entering the folder it will remain active when you leave the folder.
Deployment with pyenv and pyenv-virtualenv
NOTE Work In Progess
The target server, or more likely Docker container, should have pyenv and pyenv-virtualenv installed. The environment variable
PYENV_ROOT should be set to a location within the app folder
The various bash profile files should point to the same shims folder and include the usual
pyenv init and
pyenv virtualenv-init commands.
TL:DR - quick reference sheet
Assuming you have followed the instructions above and installed pyenv and pyenv-virtualenv properly:
To add another version of Python
pyenv install 3.6.1 pyenv rehash
List the installed versions (and environments)
List the virtualenvs
To create a virtualenv for a project
pyenv virtualenv 2.7.14 someprojectname
To activate a virtualenv
pyenv activate someprojectname
To activate the virtualenv when you enter a (project) folder
cd /path/to/your/project/folder/ pyenv local someprojectname
and deactivate that environment
To remove a virtual env
Useful if you installed the wrong version of Python for some reason
pyenv uninstall someprojectname