Python on Managed Linux Workstations¶

With the switch from Ubuntu to Debian on our managed Linux workstations, we also re-evaluated how to install and manage Python packages. We opt for a central installation with a given set of packages, instead of varying setups from one host to the other.

System Python¶

Debian Bookworm ships with Python 3.11 and the PEP 668 flag that it's "externally managed". This protects the system-wide packages by preventing the use of pip to install packages globally or within the user scope. Instead users should create proper virtual environments. We only install the minimum of Python packages required for system administration. If you need Python 3.11, feel free to create virtual environments with your packages. But in general we recommend to use our Pyenv Python.

Pyenv Python¶

We manage a central Python 3.12 installation with pyenv and hundreds of pre-installed packages in our /opt/software/ NFS mount available on all managed Linux workstations.

In order to use it, source the corresponding environment variables.

source /opt/software/pyenv/env/default

Afterwards start using Python and its packages.

python -m pip list  # list all installed packages

Note that your Python scripts should have the generic #!/usr/bin/env python3 shebang to allow easy switching of the Python interpreter.

If you need additional packages, or different versions than what we provide, create a virtual environment that inherits our installed packages, so that you only need to install the differences.

cd path/to/your/project
python3 -m venv --system-site-packages .venv
source .venv/bin/activate
pip install somepackage==1.0
pip list --local

There is currently only one installation inside /opt/software/pyenv/, but we will probably add different versions in the future. Although this is similar in spirit to what is done on Euler HPC, we may not completely freeze all package versions but potentially update some packages in-place. So consider working with dedicated venvs if your projects are very sensitive to package versions.

Configuring pyenv as additional kernel in JupyterHub¶

If you are using a locally installed JupyterHub, you can add the pyenv Python as separate iPython kernel.

source /opt/software/pyenv/env/default
ipython kernel install --name "opt-software-pyenv" --user

This creates a custom configuration ~/.local/share/jupyter/kernels/opt-software-pyenv/kernel.json in your home. Inside JupyterHub you should now be able to pick this kernel in the Kernel -> Change kernel menu.