Python Virtual Environments: The best workflow

When I start a new Python project I use pyenv install 3.11 and pyenv shell 3.11 to install and set the Python version (here to 3.11) and then python -m venv .venv to create a virtual environment that sits in my project folder. At last it gets activated with source .venv/bin/activate. I bundled these commands into a function, so that I only need to run mkpyvenv 3.11 and the virtual environment is created and activated.

The reasons for this workflow

Every project should have its own virtual environment, so that each project is independent.

I want to be able to control the Python version for each project, eg. use 3.8 for one project and 3.11 for another.

I want my virtual environment to be in the project folder, so that when I delete the project folder, that virtual environment is also gone. Like that my system does not get littered with virtual environments that I have long forgotten about, as conda or virtualenvwrapper does.

Another benefit is, that my virtual environment does not have a name that I need to remember. I can always activate it with source .venv/bin/activate in the project folder. VSCode automatically activates a virtual environment in a .venv folder, so I don’t even have to do that.

I want to install all my dependencies with pip and a requirements.txt or pyproject.toml file (and not with conda and an environment.yml), as often a project ends up running in a Docker container. As a Docker container is its own virtual environment, I don’t want to have to install another virtual environment manager in that docker container. Also pip is the Python standard, and conda is only common in the scientific community.

The workflow in practice

This is for MacOS.

Installation

Use brew to install pyenv

brew update
brew install pyenv

Then follow the instructions of pyenv init to load pyenv when starting a shell. For the zsh shell that is:

# Load pyenv automatically by appending
# the following to 
~/.zprofile (for login shells)
and ~/.zshrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

Then install the Python versions that you want to use. With pyenv versions you can see which ones are already installed.

pyenv install 3.8.16
pyenv install 3.11
...

Creating a virtual environment

Now let us assume we start a new project:

mkdir my_awesome_tool
cd my_awesome_tool

Now we set the Python version for the current shell session:

pyenv shell 3.11

When you run python --version you will see that the Python 3.11.1 version is used (or a newer patch version, as we have not been specific there). With which python you see that the python executable is in the .pyenv/shims directory. With pyenv which python you see that the python executable is stored in the pyenv directory /Users/fabian.hertwig/.pyenv/versions/3.11.1/bin/python.

So let us create a virtual environment in a .venv folder:

python -m venv .venv
source .venv/bin/activate

Now which python points to the virtual environment: /Users/fabian.hertwig/Projects/my_awesome_tool/.venv/bin/python. And again if you run python --version you will see that the Python 3.11.1 version is used. If you install a package, eg. pip install numpy then it will be stored in the .venv/lib/python3.11/site-packages/numpy directory.

Making shortcuts

To easily run through that process with just one command mkpyvenv 3.11, you can add the function below to your shell configuration file, e.g. .zshrc. Or you can use the awesome fig tool to create a dot file there which gets shared across all your fig installations.

mkpyvenv() {
    # Check if an argument was given
    if [ -z "$1" ]; then
        echo "Please specify a Python version to use, e.g. mkpyvenv 3.9.4"
        return 1
    fi
    PYTHON_VERSION=$1

    # Check if pyenv is installed
    if ! command -v pyenv > /dev/null; then
        echo "pyenv is not installed. Please install it, e.g. by running `brew install pyenv`"
        return 1
    fi

    # Install the python version if it does not exist
    pyenv install --skip-existing $PYTHON_VERSION

    # Create the virtual environment and activate it
    pyenv shell $PYTHON_VERSION
    python -m venv .venv
    pyenv shell --unset
    source .venv/bin/activate
}